Projects directory structure

Background

For a very long time, I was following a directory structure very similar to what many others do: there was a directly projects in my home directory, that contained a list of all projects I was or am working. Some projects weren’t trivial and contained multiple modules, but the structure of projects directly was very simple. Project by itself would hold the source code project inside.

However, the further I go with project, the more issue I have with this structuring approach:

• Where should I put a documentation? Should it be next to the source code? Should it be in git?

• What about various media that is related to the project but has nothing to do with source code?

• What about the data, which also irrelevant to the source code, but important for data analysis?

• and many other questions.

Although the questions are different, they actually about the same topic: where do I put files that are not related to the source code in any way?

Workspaces

And today I’ve realized, that the problem is due to my outdates approach to structuring files. It came from days where my responsibility was mostly coding project, and source code was a central for my work. And while the responsibilities changed, the approach stayed the same.

Nowadays, I need more than just source code for most of the projects. Except source code, I need to collect data, notes, documents, articles and posts. And instead of placing documents into $HOME/docs/$PROJECT and data into $HOME/data/$PROJECT and source code into $HOME/projects/$PROJECT, I better put everything into same folder. And I didn’t find a better name for this type of folders than workspace. 

And goal is to switch from structure

$HOME/projects
    project1/
        .git/
        module1/
        module2/
        pom.xml
        todo.txt
    project2/

to 

$HOME/workspaces
    workspace1/
        docs/
        data/
        dev/
            todo.txt
        source/
            .git/
            module1/
            module2/
            pom.xml
        charts/
    workspace2
        docs/
        ideas/
        source/

Versioning

And then I faced a new question – should I put a whole workspace under version system control? If docs and data and whatever other resources are important for the project, why shouldn’t those be versioned and kept safe and readily available using version control system, like Git?

“That is actually a good idea!” I though, “But it would be better to have different git repositories for each part of the workspace. As documents and code are changed at different times and for different reasons.”

And that’s the decision I’ve made – it is a good practice to use version control for all parts of the workspace, but do not put a whole workspace into a single repository. 

Workspace is just a collection of various resources coupled by the same topic. So it is ok to have a desire to keep workspace organized and reproducible – it should be easy to create a new workspace, that someone else can use if need. This also makes the idea of separate repositories a good one – no everyone cares about data or docs.

The desire to put a whole workspace into single repository can be too strong at different moments, mostly b/c of the reasons described in previous paragraph. And that’s where Git submodules can be helpful.

Summary

What was good before might not be good anymore. We should revisit our approaches when we change or our responsibilities change. 

The previous structure that I’ve used for project directories was good for my needs. But my needs changed and thus the structure is not anymore. 

I need more than just have a structure for the source code or single project. I need a new way to organize files, that would cover a set of documents, source code and data for one or multiple related projects. 

And my solution to it is “workspace”.

The most important SDE skill: Ownership

”Without ownership, there can be no leadership." - http://georgecouros.ca/blog/archives/3791

This post is about one of the most important leadership skills you can find: Ownership. I see it as a root of all other leadership skills. If you don't feel being an owner, you often don't feel neither responsibility, nor interest or empowerment to bring changes.

What is Ownership

Ownership is “the act, state, or right of possession something.” You can own computer, car or house. And as such, you feel an ownership to those things: you can install new software to you computer, clean it up, fight viruses, upgrade it etc. Same story is with car and house: you feel and are empowered to make changes and in your interest to make changes for better.

“Ownership is about getting something done no matter what. ” -http://www.tandemlaunch.com/ownership-versus-leadership/

There is also another definition of ownership. You take ownership when you feel responsibility for the results of your work: finishing up project, taking care of found issues, making future improvements. As an owner, I will get this project done, I will make sure it is tested and properly working, I will make sure that it delivers what my customers expects, and I will make sure that found issues will be resolved.

Ownership in Software Development

Actually, there is nothing special about being owner in software development. Development process is quite well defined: there are phases for gathering requirements, designing and planning, coding and testing, bug fixing and launching. List of artifacts is also well defined: source code, documentation, tests, metrics, list of issues and etc.

Group Ownership

Group ownership is also a shared ownership. Multiple software engineers own codebase, projects and features, specific modules or components. Usually, engineers in a team will have different level of ownership skills:

• some would just do the work and won’t care much of the result,
• some would focus on finishing single task successfully and wouldn’t care much of a whole picture
• others would care mostly about their own feature/component and wouldn’t care of a whole projects
• others go beyond and focus their attention on a whole project.

The higher ownership skills, the larger area they normally cover. I would call it an “ownership scale”.

Off topic: Ownership and Career Growth

Interesting that you can see a clear match between ownership scale and engineer’s position: junior software engineers are barely understand what is happening and focus on finishing up single task; mid-level engineers are focused on a codebase, features that they work on and, maybe, a few more aspects; senior engineers go beyond that and focus on a whole product or larger part of it.

Ownership level tells what are you focusing on as part of your everyday job and how how successfully you will be able to deliver results.

And based on this, I make a statement that Ownership skill should be an important part of promotion evaluation, same as technical skills are.

Working in a team is always harder than working on your own. Dealing with more people that are owners of the product (just like you) is even harder.

Group ownership requires to take into consideration thoughts and ideas of other owners. There is no a simple way to make a decision now. But luckily you get more than just a headache. Discussion with other owners can help find both consensus and a better way. It is because all owners are focused on improving things and successful delivery, not on own ego satisfaction.

The strategy of “my ownership is better than your ownership” is a lose-lose strategy. Such behavior will kill or suppress ownership attitude in others. As an owner who care about the product, searching for a path to success, you lose like minded people who could help you here. Demotivated ownership might even cause an opposite to ownership behavior. Real owner would grow other owners around, b/c in a team work this is the most effective way to delivery successful product.

Examples

Here I’m going to provide a list of different behaviors and try to give my opinion on ownership level.

• Mark finally finished the project he didn’t like much. And now he is not eager to work on the related bug that was found recently. I think you would agree with me, if I would say that Mark’s ownership level is not very high. First of all, Mark didn’t like the project and now attempts to skip working on issues that are result of his previous work.
• Hanna finished working on the project that she was assigned. She didn’t want to work on this project, but business really wanted it. But Hanna took the project and work it through and delivered successfully. Now business loves Hanna and has a new project for her. This is a good example of ownership. Hanna took the project she didn’t want at first, but liked eventually and could deliver it successfully.
• John was asked to help with task for project X. John was really helpful and could finish task successfully. He also fixed a few issues he had found, and helped Sarah to develop analytics module faster. Project X was released on time with John’s help. This is a great example of ownership. John was just asked to help with a task. Instead, he take a responsibility to fix issues and help Sarah with her work on the project. Don’t be like Mark, be like John.
• Lisa works only on the tasks that she was assigned. She doesn’t spend much time learning to do her work better. Lisa also doesn’t spend time to see how she could improve the project. Most of the time she spends on tasks added by others. Whenever she finishes with task, she moves next without a necessary testing. I’m not sure that I see any ownership here. Lisa is not owner: she doesn’t care about results and quality of her work.

Summary

In todays world, having a great technical skills is only one important part that makes a successful software engineer. But there is more than it. Ownership skill is another important part. It is important for engineer to get projects done successfully, improve team, processes and software.

If, after so many years improving technically, you still find you need to get better, then switch your focus to improving your Ownership skills.

Iterate Quickly

That's what others recommend

Recently I've being reading 2 different texts, and they both mentioned how high-velocity iterations are important and better than high-quality iterations.

First one, is article by Jeff Atwood, where author writes about “Boyd’s Law of Iteration”. Article is quite interesting. It introduces us into the history behind the Boyd's law of iteration. Article as well emphasizes the law itself: speed of iteration beats quality of iteration.

Second one, is latest annual later by Jeff Bezos to shareholders. In section "High-Velocity Decision Making", author also emphasizes superiority of high-velocity decisions over high-quality decisions. Interesting that Jeff Bezos finds that 70% of useful information is usually enough to make a decision. Waiting for another 20% or 30% of data might take too much time, and might be not as much beneficial.

In both cases, authors notice that making quick decisions and iterating fast can only work well if there is enough information available originally. And feedback is used to correct the course continuously during quick iterations.

In book The 5 Elements of Effective Thinking, authors also emphasize the importance to start fast. In section dedicated to making mistakes, authors encourage to not be afraid to start with mistake (if you don't know where to start), but continuously iterate incorporating received feedback and new knowledge.

Iterate Quickly in Software Development

What does it mean to iterate quickly in software development. I'm going to throw a few ideas, even though they are quite obvious:

1. Plan smaller and deliverables for each milestone:
   
   • define list of milestones
   • know deliverables for each milestone
   • plan time to receive and incorporate received feedback
   • follow the feedback, not the original plan: course-correcting is the right way to go.
2. Start testing faster:
   
   • write unit tests
   • write functional and integration tests
   • use staging for deploying latest versions
3. Deliver features faster:
   
   • if can't deliver a complete feature, then deliver part of it
   • if can't deliver to all customers, then deliver to some only, like internal or beta users
4. Sprint should take less time:
   
   • 1 month sprint is too slow in most cases
   • don't forget to make sprint retrospectives
5. Collect feedback from users continuously.

Why I am Not Sure that TDD is the Only Right Thing

Let's say you have a project to deliver feature XYZ. After analysis, design and planning, you came up with a term of 2 months: 2 weeks for scoping a work, 4 weeks for coding and 2 weeks for testing. Sounds great.

As anything else, testing has multiple perspectives. The most obvious one is verifying results of your work and keeping quality under control. The other one, less obvious, is minimizing the risks caused by numerous bugs. Third one is related to previous two: minimizing the cost of software development and support. The earlier you test, the easier it is to fix and less bugs you have later.

Lets look at two possible scenarios:

Scenario 1: No unit tests

So you don't believe in unit tests or you just don't have a time to write them. You'll better test everything manually during 2 weeks of test phase.

You have finished with your code, and now it is time to wire everything up and start testing. Oops. You found a small bug, where you just had forgotten to add "!" in your if statement. Quick fix. Easy peasy. You make a change. Your favorite build tool picks your last commit and makes a new build. After 10 minutes you have a binary. Another 5-10 minutes and you have it deployed and ready for testing.

You test it again, and... Oops. You found a small bug. Looks like you actually need to call that method first, otherwise you get NPE. Damn. What a stupid mistake?! Quick fix. Easy peasy. You make a change. Your favorite build tool picks you last commit and makes a new build. After 10 minutes you have a binary.

You test it again, and... Oops. You found a small bug. Well, I guess you see where I'm heading. 2 small stupid bugs, you already spent 1 hour. You have only 2 weeks, but you can’t even get it working for happy case, don’t mention for all other dozens of test cases.

And nothing prevents you from regressions during bug fixing.

You probably see now, what is wrong with this scenario. You think you've saved time during coding, but actually you didn't.



Unit tests are a convenient way to spread testing work throughout development phase. So you'd need to do less during testing phase, where fixing bug is already an expensive thing.

I rarely find bugs when I write unit tests. Maybe 1 in 20 tests that I had actually found and fixed a bug. But, man, this is so fast to fix a problem at this time. Everything is locally on my machine in my IDE. I don’t need to build and deploy change, neither others are blocked by me. Found, fixed, done! A tad of time here, saves hours later!

Scenario 2: Write unit tests during coding

So by now you've decided to try out writing unit tests. Maybe this will help to avoid the hell you've had during Scenario 1. You start writing unit tests for most of the code you produce. It has been really hard at first, but later became easier. You use mocking a lot and thus unit testing becomes almost painless. Actually, you can't imagine now how to write new code without tests, but time to time you skip writing tests for less important parts.

You've finished you coding by now, and ready to start testing. And you find that there are not that much bugs now. And when you find a bug, you know how quickly to fix it and verify with unit tests. Thus, you don't need to go through tedious fix-build-deploy-verify cycle as you used to do during Scenario 1.



As result, you spend more time during coding phase, as you need to write unit tests. You however spend less time during testing phase, as you can iterate bug-fixing faster.

Of course, you still spend a tad more time than planned: but it is because you need to learn how to estimate better!

TDD vs Plan Ahead

Assume we have 3 approaches:

1. you follow TDD and write tests first, and then write code to make your tests pass eventually
2. you take a paper or text file, and you plan ahead by writing a list of test cases that need to be converted to automated test suites; then you code and write test cases during coding or afterwards
3. you don't plan what test cases you have, you just write a code and cover it sporadically with unit tests

It is probably clear that 3rd approach is the lacking one here. You start writing code without defining what are your expectations. You don't have a plan of what needs to be tested. As result, it is easy to miss many important test cases.

1st and 2nd approaches sound very similar, as they both have the same goal: know and plan what you're going to do before you start coding.

However, 1st approach forces you to build constraints for you and your code before you start coding. Of course, you can refactor those later, again and again. While 2nd approach gives you a flexibility to write the code and refactor it until it fits your vision. You can now create tests during or after you've finished with piece of code.

Both 1st and 2nd approach require self-discipline, but in different ways. TDD needs a discipline to start and keep to using it. 2nd approach needs a discipline to convert test cases on paper to unit tests in code.

Not obvious difference between 1st and 2nd approaches is completeness: it is easy to miss something when you write a code. If you start defining a list of test cases on paper, it is always easier to identify completed list.

Summary

Here are some bullet points as a summary from this post:

• writing unit and functional tests is important to keep quality under control, save time and money
• it is never too late to write tests for the code, but now is better than later
• thus, write tests during coding and before commit and code review
• writing tests before code does not make things cheaper or better, unless you don't have a discipline to cover functionality with tests
• we need to work on improving our estimation skills.