As a passionate outside observer, I like to try and understand what the different developers do. However, despite MJ writing very thorough work reports (example here), I struggled to grasp what his work is about.
If you’re in a similar boat, then this article is for you.
Reading those reports, I felt like someone who sees the below t-shirt for the first time, but doesn’t know how base-2 counting works. I could read all the text – but it didn’t make sense!
Normally it’s fairly easy to grasp what developers and researchers do. For example:
Howard Chiu – creator of the Lightning Memory-Mapped Database Manager (LMDB), which is used to store and serve Monero’s blockchain. More recently he was involved in creating Monero’s ASIC resistant RandomX mining protocol, working in close collaboration with developer Tevador.
Sarang Noether – a Monero Research Lab member who’s played a key role in the implementation of cryptography such as Bulletproofs, DLSAG and Dandelion++
Moneromooo – an anonymous, but prolific C++ developer who has, by far, written the most active code in the Monero project.
To borrow an American football analogy used by a Redditor – the guys mentioned above are like the wide receivers and running backs of the team – scoring touchdowns and becoming well known. Using that analogy, we can say that there are lesser known developers who are like the offensive lineman – they don’t get the glory and fame – but they still play an important role
So after some back and forth messaging on IRC, MJ agreed to a phone call to discuss what he does. This article is an attempt to explain in layman’s terms, what he’s up to!
MJ-XMR’s Past Work
MJ has primarily been working on optimizing the tools that the developers use to modify and build the project.
So what does this mean? It’s quite meta.
To make sense of this, let’s first take a step back and look at how the Monero software gets built. There are a few different steps:
- In order for people globally to all collaborate on Monero, we/they use a protocol called Git and a hosted Git service called Github.
- For a user to contribute, they will copy the source code to their computer, and then start making changes.
- Once a change is made, the user then needs to compile the source code into an executable file, and test it’s working correctly.
- Once working, they will upload the change to Github, and request that it gets merged in to the main branch (called a Pull Request or PR).
- Lastly, that pull request will have Continuous Integration (CI) tests run on it, to check it doesn’t break anything critical.
Of these steps, MJ has primarily focused on speeding up the compilation stage and optimizing the continuous integration stage.
Why would this be important? Let’s take compilation as an example. Repercussions if compilation time is slow include:
- Developers avoiding compiling and testing often. Instead they may wait until a number of changes have built up. This is problematic if a bug occurs, as they have to figure out which of their multiple changes caused it.
- Long delays affect flow. If you’ve got to wait a long time for compilation before you can test your change, you can easily get distracted by other things – ultimately reducing the # of productive hours in a day.
Whilst established developers will know which specific tests to run for which change (avoiding running all the tests), they will at times still need to do a full recompilation, and thus this benefits them directly.
^ This diagram, adapted from here, represents a basic outline of the Monero development workflow. MJ’s work on compilation and continuous integration has an impact each time tests are run.
Can’t existing Monero developers optimize this stuff?
The way MJ explained it to me, is that the Monero project has been developed by a number of highly specialized developers, many of who are experts in specific fields like cryptography, distributed computing, etc. However, optimizing compilation time may not be something they focus on. He has seen this issue in other C++ projects he has worked on in the past, not just Monero.
Improving Compilation Speed
MJ’s first contribution to the project was to integrate a compiler cache called Ccache. This speeds compilation up by caching previous compilations, then detecting when the same compilation is being done again, and using that cached binary, rather than compiling from scratch.
This brought the Travis continuous integration compilation down from approximately 22 to 2 minutes for each build (source).
^ Click to see in higher resolution – it demonstrates the Ccache speed increase mentioned above
He then went on to make further continuous integration optimizations – 2 examples of which are included below:
- Shortened ubuntu-test from 2hrs to 1hr 25mins – pull #7489
- Shrinked caches stored on Github, which reduced the frequency they’re purged and have to be rebuilt from scratch – pull #7780
– Fixed issue where building Mac OSX with an M1 CPU would fail – Pull #7435
– Helped in bringing Monero to Rpi4, 64-bit – Pull #7953
– UI problem under Linux Mint & Void distros. MJ didn’t fix this, but helped in organizing the OP’s thoughts, tested and described how to reproduce the problem – Pull – #7862
– Fixed build issue. Reported by a user, who wanted to remain anonymous – Pull #7902
– Fixed EasyLogging++ issue causing application crash – Pull #7828
– Also EasyLogging++: Added Utests for Moo’s previous fix – Pull #7771
Additional Areas of Focus
Hopefully the above gives an idea of the primary things MJ-XMR has been working on. Below are some further things:
Each time a pull-request is submitted, the code needs to be reviewed by others to make sure it does the intended job, and doesn’t break anything in the process. As you can imagine, not everyone wants to be reviewing other people’s code. But helping with code reviews does help move the project along. Currently selsta manages the Github merge queue, and it’s helpful to selsta to have more help with code reviews, so PRs can get merged quicker.
Granted, MJ’s still fairly new to the project, so without deep codebase knowledge he can’t yet help with the more complex changes.
^ Example of mj-xmr and ndorf approving a pull request after review (link)
Supporting new developers
Helping onboard new developers is useful, and it’s something MJ helps with. He has done this in one-to-one interactions…
For example, on his recent CCS proposal it was nice to see JBerman’s comment: “Also very much so appreciate @mj’s deep knowledge of C++”
He has also aided with this through improving documentation, for example:
- Documentation on compiling, debugging and testing efficiently – Pull #7666
- Documentation on reproducible builds (Gitian) – Pull #7658
MJ explained that he actually has a number of further optimizations he’d like to make to the project, of which a few include:
- Refactoring wallet2
Currently in MJ’s tests (see here), the wallet2.cpp file is the most RAM intensive when it comes to compilation.
This is a problem for machines with smaller amounts of RAM. As an example, the RPi4 board with 4 cores and 4GB of RAM is only able to use 1 core to avoid crashing or heavy swapping – which could damage its SD card.
(The above RAM report is just one of the reports which MJ generates regularly after each set of merges into Monero. They are all publicly available here: http://cryptog.hopto.org/monero/health/. Having all the past reports facilitates tracking RAM use or compilation speed over time.)
What MJ would like to do with wallet2.cpp, is to refactor the code to break it up into smaller files, all the while maintaining the existing functionality and avoiding breaking anything.
This would aid in 2 key ways:
- It would reduce the amount of RAM used by machines when compiling the project.
- Smaller files would be more approachable from a development perspective. Especially for new developers.
For the time being, he says this non-invasive, yet related change has not been accepted yet – pull #7679.
- Adding forward declarations to the header files
This helps breaking “cascading dependencies”, leading to:
- Shorter compilations
- Fewer recompilations
MJ proposed this in a PR, however it caused really small changes in around 60 files (which is a lot to review) it hasn’t been accepted. However, there’s still an opportunity going forward to make this improvement.
- Adding the Unit Tests for Mac & Windows to the Github Actions workflow
Currently, each time a pull request is merged into a branch (master or other), there is a Github workflow that performs the compilation steps required for the Linux, Mac & Windows binaries. This ensures that the changes do not break the build process. Additionally, there are a series of tests (core and unit) to test certain important functionality isn’t broken – however they are only run on the Linux build.
The rationale for only running the tests on the Linux build is to save on time and server resources. With the hope being that, since the code is written in a standard and portable way, it will behave the same on all operating systems. Unfortunately this is not always the case. Currently the core team relies on users submitting issues and bugs, rather than being able to potentially detect them before they occur on MacOS & Windows.
Currently the unit tests take around 10 minutes to run (see screenshot below). If unit tests for MacOS and Windows were added, they would be run in parallel, so are unlikely to add any extra time to the tests, but would use slightly more resources.
MJ-XMR’s Future Work
Going forward, MJ will continue with his existing work, but in a lesser extent, as he will be heavily working on a major project called OSPEAD:
A Statistical Analysis Simulator for OSPEAD
Recently, a new contributor to the Monero community, Rucknium, has put forward the idea of improving the decoy selection algorithm used by ring signatures.
Currently, Monero outputs use a ring of 11 signatures, of which 10 are decoys, which are “randomly” selected from past outputs in the blockchain. Rucknium has been reminding the community (many of whom were already aware) that this algorithm for selecting decoys has statistical weakness and needs improving.
As things stand, a chain analysis company may be able to attempt educated guesses, based on repeated patterns in the data, which of the output signatures are the decoys and which is the real spend signature. Of course, we don’t know for sure.
It’s a problem that needs a statistician’s input to resolve, which the project hasn’t really had since the original research by Moser et al. (2018), and thus it’s great that it’s getting some attention now.
As part of that project, Rucknium has brought together a “team” of developers to help solve the problem. Those include JBerman, UKoeHB and mj-xmr.
MJ’s role in this will be to provide software and support for an application that allows Rucknium to run simulations on new algorithms, and check for any weakness.
This should help to avoid over fitting – which is where an algorithm is well adapted for a sample set of data, but performs badly in the real world.
Perhaps a natural question then, would be – how would Rucknium perform his work without mj-xmr’s tool? This was asked in a recent Monero-Research-Lab meeting (log link):
Which is interesting – it suggests that part of the value will be derived in the ability to collaborate together and have a second pair of eyes on the problem. I don’t know for sure, but time-series analysis of data appears to be a niche area that not every Monero developer will be able to assist Rucknium with.
Other benefits of the collaboration include:
- Focus on the speed of the simulation, reducing the risk of trade-offs between fitness of the model and time spent on training it.
- Fast communication and short turn around times between the producer (MJ) and the customer (Rucknium). Something that may be hard to achieve with other software.
OSPEAD is a very important project for Monero, as it aims to improve the weakest part of Monero’s privacy – and I’m excited to see how it progresses.
Prior Development Experience
Whilst MJ is choosing to separate his “real life” identity from his digital identity in the Monero project, he has shared some info about his technical background:
- He’s been working as a developer for the past 13 years.
- He holds a master’s degree in Computer Science and specializes in coding in object oriented languages (mainly C++, which is the language used by the Monero project).
- To pay the rent, he’s worked in various fields, from process automation, geographic information system mapping (GIS) apps, using computer vision for navigation and many other kinds of projects which involved mathematics, good design or computer resource management.
- As a side project, he has developed a couple of space flight navigation instruments in C++ for a realistic freeware space flight simulator. His current side project is an automated trading and backtesting platform, also in C++.
How he became interested in Monero
MJ started off as someone interested in the wider cryptocurrency space, doing some at-home Bitcoin mining.
However, it soon became apparent to him that ASIC mining was wasteful in terms of hardware. With each generation of new Bitcoin miners (ASICs) you had a small time window in which to recoup your investment and turn a profit, before your ASIC inevitably gets superseded by faster hardware. At this point your ASIC becomes trash, and your profit margin severely decreased if you buy the replacement ASIC.
With RandomX on the other hand, it enables users to mine Monero with CPUs. This is hardware that doesn’t become completely obsolete as soon as the next generation comes round. If you have to retire a CPU from mining, it’s still useful for a variety of tasks.
It’s interesting how different people find their way to Monero for different first reasons. I believe Sergei Chernykh (sech1), creator of P2Pool for decentralized mining mentioned on his MoneroTalk interview that he also got into Monero due to RandomX.
^ Sergei Chernykh during his Monero Talk interview
Hopefully the above provides a useful glimpse into what one of the newer developers on the Monero project has been working on.
No doubt as he continues to work on the project and becomes more familiar with it, he will increase his capacity and the value he can bring.
If you’ve any questions, please leave them below in the comments.