Hey there!
I’m a chemical physicist who has been using python (as well as matlab and R) for a lot of different tasks over the last ~10 years, mostly for data analysis but also to automate certain tasks. I am almost completely self-taught, and though I have gotten help and tips from professors throughout the completion of my degrees, I have never really been educated in best practices when it comes to coding.
I have some friends who work as developers but have a similar academic background as I do, and through them I have become painfully aware of how bad my code is. When I write code, it simply needs to do the thing, conventions be damned. I do try to read up on the “right” way to do things, but the holes in my knowledge become pretty apparent pretty quickly.
For example, I have never written a class and I wouldn’t know why or where to start (something to do with the init method, right?). I mostly just write functions and scripts that perform the tasks that I need, plus some work with jupyter notebooks from time to time. I only recently got started with git and uploading my projects to github, just as a way to try to teach myself the workflow.
So, I would like to learn to be better. Can anyone recommend good resources for learning programming, but perhaps that are aimed at people who already know a language? It’d be nice to find a guide that assumes you already know more than a beginner. Any help would be appreciated.
It’s always good to learn new stuff but in terms of productivity: Don’t attempt to be a programmer. Rather attempt to write better research code (clean up code, revision control, better commenting, maybe testing…)
Rather try to improve cooperation with programmers, if necessary. Close cooperation, asking stupid questions instead of making assumptions etc. makes the process easy for both of you.
Also don’t be afraid to consult different programmers since beyond a certain level, experience and expertise in programming is vastly fragmented.
Experienced programmers mostly suck on your field and vice versa and that’s a good thing.
Odd take imo. OP is a programmer, albeit perhaps not a very good one. Did a PhD (computational astrophysics), been working as a professional dev for 10 years after that. Imo a good programmer writes code that solves the problem at hand, I don’t see that much of a difference between the problem being scientific or a backend service. It doesn’t mean “write lots of boilerplate-y factories, interfaces and other layers” to me, neither in research nor outside of it.
That being said, there is so much time lost in research institutes because of shoddy programming by researchers, or simply ignorance, not knowing a debugger exists for instance. OP wanting to level up their game would almost certainly result in getting to research results faster, + they may be able to help their peers become better as well.
25 years in the industry here. As I said there’s nothing against learning something new but I doubt it’s as easy as “leveling up”.
Both fields profit a lot from experience and it’s as much gain for a scientist do become a software dev as an architect becoming a carpenter. It’s simply not productive.
there is so much time lost in research institutes because of shoddy programming
Well, that’s the way it is. Scientific code and production code have different requirements. To me that sounds like “that machine prototype is inefficient - just skip the prototype next time and build the real thing right away.”
My advice comes from being a developer, and tech lead, who has brought a lot of code from scientists to production.
The best path for a company is often: do not use the code the scientist wrote and instead have a different team rewrite the system for production. I’ve seen plenty of projects fail, hard, because some scientist thought their research code is production level. There is a large gap between research code and production. Anybody who claims otherwise is naive.
This is entirely fine! Even better than attempting to build production quality code from the start. Really! Research is solving a decision problem. That answer is important; less so the code.
However, science is science. Being able to reproduce the results the research produced is essential. So there is the standard requirement of documenting the procedure used (which includes the code!) sufficiently to be reproduced. The best part is the reproduction not only confirms the science but produces a production system at the same time! Awws yea. Science!
I’ve seen several projects fail when scientists attempt to be production developers without proper training and skills. This is bad for the team, product, and company.
(Tho typically those “scientists” fail to at building reproducible systems. So are they actually scientists? I’ve encountered plenty of phds in name only. )
So, what are your goals? To build production systems? Then those skills will have to be learned. That likely includes OO. Version control. Structural and behavioral patterns.
Not necessary to learn if that isn’t your goal! Just keep in mind that if a resilient production system is the goal, well, research code is like the first pancake in a batch. Verify, taste, but don’t serve it to customers.
You can learn the basics using any beginner’s course on udemy.
Then I’d recommend that you build smaller projects to practice. In the future you can change the types of projects you’re building to practice even more (rest apis, website frontend, messaging app, low level stuff with C/rust/golang), if you want yo reach this level, of course.
I’ve got two tips to add to the pile you’ve already read.
I recommend you read the manuals related to what you are using. Have you read the python manual? And the ones for the libraries you use? If you do you’ll definitely find something very useful that you didn’t know about.
That and, reread your code. Over and over until it makes total sense, and only run it then. It might seem slow, and it’ll require patience at first. Running and testing it will always be slower and is generally only useful when testing out the concept you had in mind. But as long as you’re doing your conceptual work right, this shouldn’t happen often. And so, most work will be spent trying to track down bugs in the implementation of the concept in the code. Trust me when you read your code rigorously you’ll immediately find issues. In some cases use temporary prints. Oh and avoid the debugger.
As one physicist to another, the most important thing in the code are long variable names (descriptive) and comments.
We usually do not do multi-people multi year projects, so all other comments in this page especially the ones coming from programmers are not that relevant. Classes are cool, but they are not needed and often obscure clarity of algorithmic/functional programming.
S. Wolfram (creator of Mathematica) said something along these lines (paraphrasing) if you are writing real code in Mathematica - you are doing something wrong.
We usually do not do multi-people multi year projects
Seriously - why not?
Say you’re doing an experiment, wouldn’t it be nice if someone else could repeat that experiment? Maybe in 3 years? in 30 years? in 3,000 years time? And maybe they could use your code instead of writing it themselves and possibly getting it wrong?
If something is worth doing, then it is worth doing properly.
Classes are cool, but they are not needed and often obscure clarity
I write code all day professionally. A lot of my code doesn’t use classes. I agree they often “obscure clarity”.
But sometimes they do the opposite - they make things crystal clear. It’s important to know how to use classes and even more important to know when to use them. I guarantee some of the work you do could benefit from a few simple classes. They don’t need to be complex - I wrote a class the earlier today that is only four lines of code. And yes, a class was apropriate.
The reason they don’t do multi people and multi year coding projects has nothing to do with repeatability of the experiments, most science coding is done through simple-ish code that uses existing libraries, doesn’t code them. That code is usually stored in notebooks (jupyter, zeppelin) or simple scripts.
For science code, it usually falls in the realm of data analysis, and as a data engineer, let me tell you that the analysis part of the job is usually very ad-hoc modifications of the script and live coding though notebooks and such.
The part where whatever conclusion of the research is then transformed into a functioning application, taking care of naming conventions, the architecture of the system where the input data, the transformations, the postprocessing and such is done, is usually done by another team of dedicated data engineers or software developers.
I guess that it would be helpful for the analysis part to have standardized templates for data extraction and such, but usually the tools used in the research portion of the process and the implementation portion are completely different (python with tensorflow vs C++ with openvino or whatever cloud based) so it’s not really fair to load the architecture design since the beginning.
You know how changing requirements is the bane of real™ production grade™ software?
In science requirements change all the time. You write some 50-100 lines to plot your results. You realize that the effect is not visible, so you go back to the lab, change 5 variables and run the test again. Some quick code changes and you see the effect. Perfect. Now you do the measurement as a function of temperature. You adapt your script, you indent the data processing code to turn your list of files into a list of characteristics parameters and adapt the plotting. You run the experiment for the three samples you have prepared and compare their plots. Some more experiments tuning and corresponding script tuning is required. You take the characteristic parameters that your code (grew to 200 lines now, but whatever) calculated and write a new script to take that array and plot it nicely.
Now someone wants to repeat the experiment 4 years later. The measurement equipment changed and the data format is slightly different. It’s impossible to document the exact state of the hardware in your code anyway. They are actually interested in a different effect and want to plot that as well, but they need the effect/characteristics parameters that are shown by your code as a sanity check. They need to rewrite 125 of the 200 lines.
There never is a finished product that is worth maintaining long term. Everyone using the script has to understand the domain precisely anyway. Is it worth it to reuse the old code when you need to rewrite more than half of it anyway?
Don’t get me wrong, code reuse does happen. But it’s much more “oh, I wrote that three months ago somewhere else” ctrl-c ctrl-v. “Ok, now I need to change five lines in this function to adapt to the new thing I’m trying to do.” It makes absolutely no sense to write that function in a abstracted way. Every time you use it the requirements changed and the abstraction is no longer valid anyway.
Great potatoes… This is not very good advice. Ok for prototypes that are intended to be discarded shortly after writing. Nothing more.
Yes, those prototypes are the goal here.
Learning new programming languages is an awesome way to expand your programming brain. If you want to stay in the same scientific computation niche, you can check out Julia or Mathematica. If you’re just looking to broaden your horizons, the world is your oyster. For me, learning Clojure really cooked my noodle but made me a much better programmer since it taught me functional programming.
Also, just read other peoples code! You can learn the conventions that way. Though for you it would best to find other products within your niche, because I’m not sure if general web dev code would be super helpful.
There are techniques that are broader than any single language’s conventions, and I think learning those are how you can improve. That’s hard to teach, though, and it comes from experience with a few different languages, in my opinion.
And honestly, I can totally respect the “conventions be damned” attitude, because at the end of the day, you’re trying to make something that works, and if nobody else is reading that code, you’ve made the right trade-off.
Forget everything you hear about OOP and just view it as a way to improve code readability. Just rewrite something convoluted with a class and you’ll se what they’re good for. Once you’ve got over the mental blockade, it’ll all make more sense.
To add to this, there are kinda two main use cases for OOP. One is simply organizing your code by having a bunch of operations that could be performed on the same data be expressed as an object with different functions you could apply.
The other use case is when you have two different data types where it makes sense to perform the same operation but with slight differences in behavior.
For example, if you have a “real number” data type and a “complex number” data type, you could write classes for these data types that support basic arithmetic operations defined by a “numeric” superclass, and then write a matrix class that works for either data type automatically.
One is simply organizing your code by having a bunch of operations that could be performed on the same data be expressed as an object with different functions you could apply.
Not OP, but also interested in wrapping my head around OOP and I still struggle with this in a few different respects. If what I’m writing isn’t a full program, but more like a few functions to process data, is there still a use case for writing it in an OOP style? Say I’m doing what you describe, operating on the same data with different functions, if written properly couldn’t a program do this even without a class structure to it? 🤔
Perhaps it’s inelegant and terrible in the long term, but if it serves a brief purpose, is it more in the case of long term use that it reveals its greater utility?
I use classes to group data together. E.g.
@dataclass.dataclass class Measurement: temperature: int voltage: numpy.ndarray current: numpy.ndarray another_parameter: bool def resistance(self) -> float: ... measurements = parse_measurements() measurements = [m for m in measurements if m.another_parameter] plt.plot( [m.temperature for m in measurements], [m.resistance() for m in measurements] )This is much nicer to handle than three different lists of temperature, voltage and current. And then a fourth list of resistances. And another list for
another_parameter. Especially if you have more parameters to each measurement and need to group measurements by these parameters.Say I’m doing what you describe, operating on the same data with different functions, if written properly couldn’t a program do this even without a class structure to it? 🤔
Yeah thats kinda where the first object oriented programming came from. In C (which doesn’t have classes) you define a struct (an arrangement of data in memory, kinda like a named tuple in Python), and then you write functions to manipulate those structs.
For example, multiplying two complex vectors might look like:
ComplexVectorMultiply(myVectorA, myVectorB, &myOutputVector, length);
Programmers decided it would be a lot more readable if you could write code that looked like:
myOutputVector = myVectorA.multiply(myVectorB);
Or even just;
myOutputVector = myVectorA * myVectorB;
(This last iteration is an example of “operator overloading”).
So yes, you can work entirely without classes, and that’s kinda how classes work under the hood. Fundamentally object oriented programming is just an organizational tool to help you write more readable and more concise code.
All the other comments are great advice. As an ex chemist who does quite a bit of code I’ll add:
Do you want code that works, or code that works?! It’s reasonably easy to knock out ugly code that only works once, and that can be just what you need. It takes a little more effort however to make it robust. Think about how it can fail and trap the failures. If you’re sharing code with others, this is even more important a people do ‘interesting’ things.
There’s a lot of temporary code that’s had a very long life in production, this has technical debt… Is it documented? Is it stable? Is it secure? Ideally it should be
Code examples on the first page of Google tend to work ok, but are not generally secure, e.g doing SQL queries instead of using prepared statements. Doesn’t take much extra effort to do it properly and gives you peace of mind. We create sboms for our code now so we can easily check if a component has gained a vulnerability. Doesn’t mean our code is good, but it helps. You don’t really want to be the person who’s code helped let an attacker in.
Any code you write, especially stuff you share will give you a support and maintenance task long term. Pirate for it!
Code sometimes just stops working. - at least I’m my experience. Sacrifice something to the gods and all will be fine.
Finally, you probably know more than you think. You’ve plenty of experience. Most of the time I can do what I need without e.g. classes, but sometimes I’ll intentionally use a technique in a project just to learn it. I can’t learn stuff if I don’t have a use for it.
I’m still learning, so if I’ve got any part of the above wrong, please help me out.
“Pirate for it” was probably the wrong phrase. “Plan for it” was probably what you were thinking when your fingers did something else.
Thought I did so well on my phone. It kept auto correcting code to coffee. Maybe it was telling me something.
Yes, plan for it!
The thing to think about is reusability. Are you copying and pasting code into multiple places? That’s a great candidate to become a class. If you have long lived projects (i.e. something you will use multiple times over a lot of years) maintainability is important. Huge functions and monolithic applications are very hard to maintain over time.
Break your functionality out into small chunks (methods and classes). Keep it simple. It may take a while to get used to this, but your time for adding additional functionality will be greatly improved in the long run.
A lot of great programmers were terrible at one time. Don’t let your current lack of knowledge of principles stop you from learning. One of the biggest breakthroughs I had as a programmer is changing how I looked at architecting applications. Following SOLID principles will assist a lot in that. Don’t try to understand and use these principles all at once, take your time. Programming isn’t what you make your living with, it’s a tool to help you be more efficient in your current role.
Realize that becoming a more effective programmer is different for everyone. Like you, I was self taught. I was a systems and network engineer that decided to move into software development. I’ve since moved into a role that takes advantage of all the skills I’ve learned through the years in SRE. like you, a lot of what I write now is about automation and analysis.
Careful with this. Not everything needs to be reusable, and copy/paste isn’t inherently bad.
You aren’t wrong… But everything with extended use needs to be maintainable. Making a change in 5 places sucks.
Plus, that’s what open-closed principle is all about. Instead of adding additional functionality to current working code, you extend and modify.
Making a change in 5 places sucks, making it in 2 could be reasonable. If 2 pieces of code are similar but different enough, I’ve seen way too often people try to force them into a common abstraction. That’s more what the article is about.
Check Udemy for courses and wait for a sale. They normally list for hundreds of dollars but routinely (pretty much monthly) for about $10 - $15 dollars.
Can anyone recommend good resources for learning programming
Honestly? No. The best resource is you. Ask questions. Get experience. Ask questions. Get experience. Repeat.
It’s not enough to learn. You also have to do. And you really should learn by doing in this field.
First of all - fuck Python. I’m sure it’s possible to write good code in that language, but it’s not easy and it requires a lot of discipline. I don’t mean to be mean to Python, it’s a truly wonderful language, arguably one of the best languages, when used properly. But it sounds like you’re not using it properly.
Pick a language that:
- Has static typing
- Does not do garbage collection
Static typing forces you to have more structure in your code. You can have that structure in a dynamic language but nobody ever does in practice and part of the reason is all of the libraries and third party code you interact assumes you have dynamic typing as a crutch to quickly and easily solve hard to solve problems.
It’s far better to actually solve those problems, rather than avoid them. The result tends to be code where bugs are caught when you write the code instead of when someone else executes the code. That “you vs someone else” distinction is a MASSIVE time saver in practice. It took me about 20 years, but I have learned dynamic typing sucks. It’s convenient, but it sucks.
For more info: https://hackernoon.com/i-finally-understand-static-vs-dynamic-typing-and-you-will-too-ad0c2bd0acc7
On garbage collection - it’s a similar issue. It’s really convenient to write code where “someone else” deals with all that memory management “garbage” for you but the reality is you should be thinking about memory when you write your code because, at it’s heart, 99.999% of the code you write is in fact just moving memory around. Garbage collection is like “driving” a Tesla with autopilot active. You’re not really driving at all. And you can do a better job if you grab that wheel and do it yourself.
I recommend starting with a manually memory managed language (like RUST) to learn how it works, and then from there you might try a language that does most of the work for you without completely taking it out of your hands (for example Swift, which has “automatic” memory management for common situations but it’s not a garbage collector and in some edge cases you need to step in and take over… a bit like cruise control in a car if we’re going to use that analogy.
That’s enough for now. Just keep muddling about learning those languages first before trying to tackle bigger problems. Programming is a difficult task, just like a baby learns to sit up, then roll over, then crawl, then stand, then walk with assistance, then stumble around, then walk, then run, then ride a bicycle with three wheels, then a two wheel one with no pedals, then a bicycle with pedals, then a car after that…
You skipped all those steps and went straight to driving a car (with autopilot). To learn properly, you don’t need to go all the way back to “sitting up and crawling”, but you should maybe go back just a little bit. Figure out how to get code to run, at all, in a language like rust, get familiar with it.
After you’ve done that come back here and ask what’s next. We can talk about SOLID, Test Driven Development, all the intricacies of project management in git, exceptions vs returning an error vs failing silently, and when to use third party code vs writing your own (phew boy that’s a big one…).
But for now - just learn a lower level language. Programming is a bit like physics. You’ve got elements, and under that atoms, and under that… well I don’t even know what’s under that (you’re the scientist not me). There are “layers” to programming and it’s important to work at the right layer and it’s also important to understand the layer below and above the one you’re working at.
If Python is at layer x, then you really need to learn layer x-1 in order to be good at Python. You don’t need to go all the way down - you can’t go all the way down (how do magnets work?).
Read your own code that you wrote a month ago. For every wtf moment, try to rewrite it in a clearer way. With time you will internalize what is or is not a good idea. Usually this means naming your constants, moving code inside function to have a friendly name that explain what this code does, or moving code out of a function because the abstraction you choose was not a good one. Since you have 10 years of experience it’s highly possible that you already do that, so just continue :)
If you are motivated I would advice to take a look to Rust. The goal is not really to be able to use it (even if it’s nice to be able able to write fast code to speed up your python), but the Rust compiler is like a very exigeant teacher that will not forgive any mistakes while explaining why it’s not a good idea to do that and what you should do instead. The quality of the errors are crutial, this is what will help you to undertand and improve over time. So consider Rust as an exercice to become a better python programmer. So whatever you try to do in Rust, try to understand how it applies to python. There are many tutorials online. The official book is a good start. And in general learning new languages with a very different paradigm is the best way to improve since it will help you to see stuff from a new angle.
How to think like a computer scientist may help.
As the other commenter said, you want to learn about programming principles. Like, low coupling or don’t repeat yourself.
How long is your longest program? What would you say is a typical length?
You say your code is “bad” – in what ways? For example:
- Readability (e.g. going back to it months later so you go “oh I remember” or “wtf does this do?!”
- Maintainability (go back to update and you have to totally rework a bunch of stuff for a change that seems like it should be simple)
- Reliability (mistakes, haphazard “testing”, can’t trust output)
- Maybe something else?
As a researcher: all the professional software engineers here have no idea about the requirements for code in a research setting.
I recommend you use
- git. It’s nice to be able to revert changes without worry.
- descriptive variable names. The meaning of descriptive is highly dependent on your situation. Single letters can have an obvious meaning, but err on the side of longer names if you’re unsure. The goal is to be able to look at a variable and instantly know what it represents.
- virtual environments and requirements.txt. when you have your code working you should have pip (or anaconda or whatever) take a snapshot of your current python installation. Then you can install the exact same requirements when you want to revive your code a few months or years down the line. I didn’t do that and it’s kinda biting me in the ass right now.
As a researcher: all the professional software engineers here have no idea about the requirements for code in a research setting.
As someone with extensive experience in both: my first requirement would be readability. Single python file? Fine with that. 1k+ lines single python file without functions or other means of structuring the code: please no.
The nice thing about python is that your IDE let’s you jump into the code of the libraries you’re using, I find that to be a good way to look at how experienced python devs write code.
You can jump to definition in any language. In fact, python may be one of the worst ones, because compiled libraries are so common. “Real signature unknown” is all you will get some times. E.g. Numpy is implemented in C not python.
My point about the jumping into was that you can immediately start reading the sources. Most alternative languages are compiled in some form or other so all you’ll see is an API, not the implementation.
My comment was not asking for clarification, I am contradicting your claim.
Granted, my experience is mostly limited to python and rust. But I find that in python you reach the end of “jump to definition” much much sooner. Fundamental core libraries of Python are written in C, simply because the performance required cannot be reached with python alone. So after jumping two levels you are through the thin wrapper type and your compiler will give you an “I don’t know, it’s byte code”.
In Rust I have yet to encounter this. Byte code is rarely used as a dependency, because compiling whatever is needed is no issue - you’re compiling anyway - and actually can allow a few more optimizations to be performed.Edit: since wasm is not yet wide spread, JavaScript may be the best language to dig deep into libraries.
Mostly ML or data processing libraries I would assume, I’ve read tons of REST server and ORM python code for instance, none of that is written in C.
Wrt rust: no experience with that. I do do a lot of C++, there you quickly reach the end as typically you’re consuming quite a bit of libraries but the complete sources of those aren’t part of what is parsed by the IDE as keeping all that in memory would be unworkable.
