What we definitely want to do is avoid confusion and mess. Having a clear project structure and workflow has many advantages. While there is no one single correct way to organise a project, putting a bit of thought into it, and learning from past projects can help a lot. This is the workflow that works for me.

I keep all the project code and output in GitHub, using a clear folder structure. Using GitHub is very handy. One of the problems I’ve encountered in the past is complete confusion. Rapid iteration of code and testing ideas often leads to a proliferation of functions. However, if some exploratory changes don’t pan out then it’s almost impossible to go back and undo changes distributed over multiple files made since last Tuesday at 3:45pm. The ability to define ‘save points’ every time you make a GitHub commit is very useful. Being able to create exploratory dev branches is also very useful.

However, I’ve opted to use GitHub just to contain code. I only really want one current version of the manuscript. Dropbox provides a simpler version history over time if there is some mistake to fix, but I don’t find any need for branching etc that I find useful for code. I am not certain that this is the best way, but I see the paper and code as being somewhat independent.

The manuscript and explanatory figures are kept in dropbox, along with any large datasets that are not suitable for GitHub.

project-structureThe code and paper also exist on whatever local machine I am dealing with (my laptop, or office iMac). So we would have a few folders, depending upon the project: one for the project code and one for the paper. The project may rely on other code libraries, in which case they would be included in different folders as well. And if we have very large datasets, then they would be stored locally, possibly on dropbox as well.

All of this works reasonably well. The paper on dropbox is synced automatically, so there is nothing much to think about there. GitHub takes a bit of getting used to, but I personally find the GitHub desktop app (mac) is very quick and easy to use.

The only annoyance so far comes from keeping the code and paper separate. As the code creates figures and other output, I tend to manually copy updated figures into the manuscript paper. Not the end of the world, but in the future I may automate this so that copies of figures that will appear in the manuscript are copied over there.

Further reading

If you have not yet had enough, then this video by Trevor Bekolay might be of interest.

Join the conversation

1 Comment

  1. Seeing workflows of other people is always inspiring 😉
    One thing I struggle often is how to separate a code for experiment preparation (stimuli generation, protocol generation) and results/analysis/visualisation scripts. Both stages often use similar code (I use R), but if they become more complex it becomes complicated to separate the files/functions. Probably one must allocate regular time for clean-ups.
    I like git as a productivity tool: commits are like micro-achievements.

Leave a comment

Leave a Reply