September 1, 2010
Or at least the latest version seems to fix one particular case I’ve been getting. Today Walter closed an issue in bugzilla and just it might be the bug which has been tormenting all DMD-win users since about… forever. Try it yourself: OPTLINK 8.00.6 – it could be a life-saver. I think my plush mudkip and me are having a party tonight.
BTW, In case you haven’t heard, reddit is the new bugzilla vote system
Filed under: Uncategorized | Comments (3)
July 15, 2010
Disclaimer: This is a personal opinion of a long-time D 1.0 user. Every time I get asked about my experiences with D, it deranges into this. I thought I’d write it down and not re-type each time someone tosses the question. It may not reflect the actual state of D and certainly not the state by which it’s being marketed. If you’re positively emotional about D, you probably don’t want to read this. You might read http://h3.gd/devlog/?p=16 for a lighter (and older) version instead.
I knew quite a bit of C++ when I started learning D. You can never really get to know the beast completely, but I’d been around 4 years into it, feeling rather comfortable about it. Not to the point of being able to read Boost code, but enough for my daily needs ;) I also took up some Python before starting on D and knew a bit of x86 Asm.
Yet even with these traits, learning D was a long process – there are (still) no tutorials and articles to teach you some of the finer points of the language, traps and costs hidden therein. Back at the time, D was still evolving (I jumped aboard at around D 0.68) so the process involved going through the specification a few times, experimenting, reporting bugs and following the newsgroups.
After 6 or so years of using it, I’ve grown rather proficient with D and I think it allows me to be more productive than C++ ever could. Still, there are some traps and issues which made that difficult to achieve. The D language was meant to be easy to implement and the DMD compiler was supposed to be clearly designed and implemented. But as D matured, the growth happened through accretion and some concepts were glued on without much care to how they might interact at finer levels in the language as a theoretical entity as well as in the implementation.
You get a GC which is non-generational, non-concurrent and global, along with C-style (non-discriminated) unions. You get .di file generation, which is supposed to automatically provide something resembling C headers, and the feature is outright ‘borken’. There are template mixins, which are supposed to work just like copy-paste, but are far from it (yea, you can use ’string mixins’ but they’re just a huge and buggy hack anyway). Another example – the compiler will usually spectacularly poo itself when presented with cyclic imports. Not to say these are broken per se, but it might just happen that something explodes in a weird way and you get a cryptic error you can’t really fix, which would not occur without cyclic dependencies between modules.
And even if the bug gets fixed, when working on a larger project, you get to experience how poorly implemented the compilation model is – you can pass multiple .d files to the compiler and it will emit some symbols just once – e.g. template instantiations and initializers. But it will output multiple object files, and it’s not specified in which files particular symbols will land. Good luck performing incremental builds with that! And this once again leads me to D’s biggest issue – the toolchain. I needed to create my own build tool which does its best to incrementally compile projects. Walter might claim that DMD is fast, but it’s not exactly blazing when you confront it with a few hundred thousand lines of code. With C/C++, you’d split your source into .c and .h files, which mean that a localized change of a .c file only requires the compilation of a single unit. Take an incremental linker as well, and C++ compiles faster than D. With D you often have the situation of having to recompile everything upon the slightest change. The DMD frontend also takes an interesing approach to memory management – it’s based around a GC which is… disabled due to some bugs.
Another toolchain-related issue: need to profile some code written in C/C++? Just fire up VTune, CodeAnalyst, (Very) Sleepy or the Visual Studio Team Edition profiler. D on Windows? Sadly, it uses an archaic debug info format which goes into PE – the CodeView. And that in turn meant that I needed to write my own profiler. The situation on Linux is probably better, since DMD on Linux can finally generate proper debug info, since about… two months ago :P And yea, there’s built-in code-instrumentation via -profile, but I’m after a statistical profiler, not one with reduces my soft real-time app to a turtle.
On Linux you’re also lucky enough to be able to use the GCC linker, however on Windows you get to work with OPTLINK. It’s fast! It’s tiny! It’s 20 years old and will crash in your face when you feed it too many symbols. Its crash-rate was reduced slightly when Walter disabled the emission of WKEXT symbols (weak extern in OMF-parlance) from DMD, but I still need to compile parts of my projects without debug info and contract checking. I also needed to add a custom compiler pragma which suppresses symbol generation from code used only at compile-time.
But yeah, it’s not all bad – you get to use some handy meta-programming features, such as ’static if’, tuples, ’static’ foreach (loops unrolled explicitly at compile-time), compile-time function execution, as well as useful runtime constructs. Fast delegates, foreach/opApply, guaranteed initialization, handy SSE-optimized array ops, RTTI which sucks less than C++’s. There’s a layer of pretty tasty sugar on top, such as local type inference, delegate literals / lambdas, class references, lack of ‘::’ and ‘->’, built-in arrays (if you don’t care about the speed of their allocation) and a few others. In the end, these make me pretty productive and comfortable, but learning how to use the language and its tools optimally has been a painful experience. It would certainly be easier if the creators weren’t so busy flaming the Tango library (or just posting FUD and not bothering to correct it) and ignoring the (sadly, very small) community.
Users normally aren’t better, though. They try the language and expect everything to be rock solid – this has been since the beginning of D. So they play with it, decide it’s not ready yet and leave for other pastures. There are a few of us who have been hanging around for about as long as D exists. Some have tried contributing code to the standard library, which turned out impossible. Then Tango was born as an alternative standard lib. There were discussions of it becoming the standard as it de-facto is within the 1.0 community, but they were later forgotten by the upper management. And now the Tango project is the black sheep as its contributors have grown generally tired of interacting with ‘the creators of D.’
Ah yea, and now there’s the “D 2.0″ thing which is more confused than ever. Does it want to be a better C, a better Java, a more contrived Boost? I can’t tell any more.
Someone please fork D 1.0 to hell. I’m already using a modified compiler ( no built-in Thread-Local Storage in DMD1?! Despite it only requiring changes in about 3 lines of the source code? WTF. ).
Filed under: Uncategorized | Comments (9)
June 25, 2009
Jarrett can complain like no one else. But he also makes many good points about the state of D. There are tons of inconsistencies and flaws in the language and its toolchain. But that has to be quite obvious. After all, if it wasn’t for these deficiencies, D would certainly be a major player in the programming industry.
It’s true – the future of D is uncertain, but it has always been such. Has it worsened considerably in the recent future? Nope. Has it improved? Yes, but not much.
Some time ago there wasn’t even a good system library for D1. As Jarrett successfully notes, Phobos is underwhelming. I’d go even further by being politically incorrect and saying that it’s a complete mess. My recipe for spaghetti: try drawing an import graph of Phobos. It’s full of cycles and as we know, the DMD frontend hates cycles. They produce some of the most incomprehensible error messages the compiler can spew.
Fortunately, we don’t have to use Phobos anymore with D1. Tango is a much better option, as most of the community have already figured.
Not that long ago, D could only be reasonably used on two platforms: Windows and Linux. Now it can also run on OSX, FreeBSD, Solaris and there are folks that use it on Nintendo DS.
The other areas of D are also improving – QtD is great, Descent never ceases to amaze with its IntelliSense and compile-time code views, and LDC can do awesome code optimization, sometimes beating C++ and usually matching it.
I don’t agree with Jarrett on the matter of Tango – in my opinion, it’s quite stable and not hard to keep up with at all. If you’re using the SVN revisions, you’ll at least have to follow the Trac timeline for it. An RSS feed read daily will suffice just fine. But if you don’t feel like riding the cutting edge, sticking to the larger releases is what most people do anyway. In this case, you’d just have to read the ‘Breaking changes’ section when it’s out. And except the major I/O system overhaul, it never gets any lengthy. Writing a library to support both the cutting edge and the stable version might be quite tricky, but that’s what we have version blocks for.
As for the supposed difficulty configuring it, there are the effortless bundles as well as instructions for manual installation.
So D actually is improving, thus Jarrett is a madman and a drunkard. Well, no, he’s not, at least not always. You need to take his latest post with a grain of salt. The thing about frustration is that it stacks up. And folks are getting more and more frustrated with specific parts of D. As far as community goes, it’s doing fine: several folks working on a native D kernel, the first demoscene production, a few games in the making and many projects that I personally lack comprehension in order to discover fully. But unfortunately, the opinion that is becoming increasingly common is that the makers of D, most notably Walter can’t keep up. The specification is vague on many topics, to say the least. It’s not clear whether we’re really programming in D or perhaps in “DMD.languageof”. This issue is extremely efficient at depressing the makers of alternative compilers, like LDC and Dil. They would all like the corner cases to be cleaned up. Sure, they could submit patches and hopefully Walter would include them. But at one point, you can’t even know whether something is a bug or a feature. How do you fix something that is not well specified, but it’s known that the specification is frozen?
Another part that I think deserves special attention is my personal favorite: OPTLINK. The linker that you will most probably use if you program in D on Windows. That is, unless you’re adventurous and compile with a custom-patched version of GDC on top of MinGW or LDC without exceptions. The thing about OPTLINK is that it’s awesome AND it sucks. I mean, it’s very fast and we all love it for that. But at the same time, anyone doing non-trivial development with D probably knows the dreadful MessageBox Of Death. A recipe for obtaining it: build a medium-sized D project with debug info. You’ll get to know the x86 register values as OPTLINK comes crashing down at your face.
What Walter says: it’s hard for a module to reach the limit of “fixups”, it can be split up.
What happens in the real world: templates. And with them, ClassInfo, TypeInfo, debug info. Sum it all up and it’s not hard for a module to reach the fixup limit (which is an OPTLINK bug, not a problem inherent in OMF – the object file format it uses – DDL can process these ‘too large’ files just fine). It does happen in practice, and when it happens, it’s really bad. It’s grown to the status of voodoo, and in fact magic is used to prevent it from happening as well. “Try putting the modules in a different order when passing them to the compiler” and “Compile everything at once instead of one module at a time” are two such incantations. Sometimes they work – the compiler will put template instances in different modules, depending on which ones are encountered first. But automatic build tools don’t really make this well controllable. And if you’ve tried all the combinations of module order and you’re still getting the error? Remove -g when compiling. No debug info for you. But when that doesn’t suffice? You’re screwed. Like me.
I was working on ‘xpose2′, a meta-data processing and generation lib that allowed me to effortlessly expose D classes to MiniD, serialization and whatever else I wanted. Using xpose2 with MiniD and my game was quite easy and pleasant initially. That is, until OPTLINK issues struck. So I put the modules that used xpose2 into a separate package, compiled them into a lib separately from the rest of the project and built them without -g. I could lose debug info for them with relatively little disadvantage. Fine for a while, I continued binding more classes to MiniD. The linker was more persistent than that – the issues came back… But I don’t give up so easily – I split the bindings into more modules and worked with it for a while. Then it turned out DMD didn’t like me for splitting it up and gave me some strange crashes at my program’s runtime. After a lengthy investigation, it turned out it was a codegen bug related to the supposedly fixed issue of template mixins not being instantiated enough times, so e.g. typeof(this) yielded the incorrect type. After a whole process of giving up on it, doing ‘hg rm minid’, then reverting it, I had to settle on a hack that would mean I would not be screwed that much. I’ve modified xpose2 to rely more on string mixins (which I despise but have to use them in order to do the more complex metaprogramming) and added a custom pragma to DMD which would cause it not to emit symbols or debug info for functions and types used only for CTFE.
If you’d like to support DMD-win for your medium-to-large D project, these are just the issues you might face. QtD, MiniD and my yet unannounced project are just a few of these that do. The #d IRC channel holds more instances of this story.
I’ve known quite a few people using D for about 5 or more years now. They have collectively written probably a few hundred thousand lines of multi-paradigm D code that often pushes the limits of the compilers. I believe they’re an important part of the community, but I might be alone in this belief. You’d think that they have a huge impact on D because after all, Walter says that D is community-driven. I began to suspect that the community is Walter, Andrei and Bartosz. Don’t get me wrong – I wouldn’t dare to diminish their great work and all the efforts and sweat they put into making D a reality. The thing is – they don’t know or use D that much as one would think. They sure are keen on specific parts of the spec or how the compiler *should* behave, but I doubt they know all the traps and workarounds one learns about when doing some real D. Who does? The frustrated guys. They aren’t frustrated in order to upset Walter and co., but because they deeply care about the language into which they put their hopes and spent their time instead of going mainstream. I think they all realize that D has a great potential and allows to program things in ways impossible in most other languages.
Issues with .tupleof? Try doing it in C++, C# or Java. Many bugs that pop up in DMDFE are just that. We’re all grateful to Walter for enabling us to use a language with such an amazing feature set and we hate him at the same time because when forced to use other languages, we feel like slitting our wrists. But after the hundredth time something breaks in the compiler that was reported two years earlier, you begin to question the point of sticking with D. Perhaps the same could be done with an elaborate C preprocessor hack or a snippet of Bash script that will generate code as a pre-build step? We stick to D because we believe that Walter will finally clean it all up. Yet, there seems to be a disjoint vision of D by the people who use it and the people who shape it.
D2 is currently on the horizon, as well as a book about it. If I remember correctly, it’s meant to be out by the end of 2009. The new features it will have could just be groundbreaking. Sharing, proper constness, purity, enhanced meta-programming – they will each attract their share of new programmers. But if these people find that next to these features, there are some long-standing warts in the toolchain, like improper protection for renamed imports or forward reference errors declared inexistent by the specification, they’ll run away screaming… And damn me if I’m wrong, but the OPTLINK issues we’ve had are going to be nothing compared to what will come up when when people start using Phobos 2 with its template-heavy code. D needs a new toolchain and it needs it badly.
Does D have a future? I don’t know. Many people have decided that it doesn’t and left for other (greener?) pastures. The DSource graveyard is just a proof of that. Fortunately, many of the dead projects are in a shape good enough for reuse. Some good code can be still found e.g. in Core32, MinWin, Kong or heck, DSP. It takes a bit of necromancy at times, but overall, working with D can be quite productive. I’ve recently uncovered Enki2, and it’s doing a great job at parser generation for a larger project.
And as long as I can be productive with D, I like to believe that it has a future. Whether that future is Bright, is mostly up to one man to determine.
Filed under: Uncategorized | Comments (2)
November 18, 2008
There seems to be a growing demand for dynamic linking of D code in a way that would not be a complete PITA. While *nix users could work with SO to some extent (as it apparently works fine with GDC), Windows blokes are left to the mercy of DLLs. As we all know, DLLs are scary. Yet still not many folks are taking a look at DDL.
The current status is that the version in the official SVN is not very stable or maintained – the linker has some issues, updates for new DMD and Tango version aren’t folded in regularly. I’ve been maintaining a separate branch of DDL with some of the issues fixed, but most importantly, I’ve created a custom linker that seems to function much better than the one in the official DDL repo. It sits at http://team0xf.com:1024/linker/ and is released under the MIT license, thus you have no excuse for not using it.
While my presentation at the Tango Conference 2008 described the linker in some detail, it lacked step-by-step instructions for using it. This post fixes that issue.
Filed under: DDL | Comments (8)
May 21, 2008
As some of you may have read, I’ve released a beta version of my new GUI toolkit, Hybrid.
The projects section of my website has been listing Hybrid for quite some time, yet that version was merely a prototype. The current one is actually a third installment of my OpenGL-powered GUIs. Be it a photon mapping program or some interactive walk-through, I’ve always wanted to have an in-game / in-engine GUI to play with. Using external controls, such as a separate MinWin pane next to my OpenGL context just didn’t feel right. So I rolled my own.
The first one was a rather classic retained-mode GUI that could make pictures such as:
It also had a more grayish look. Nevertheless, what pissed me off the most about its programming model, was that I had to create tons of wrappers, ‘GUI action handlers’ and separate the program into GUI init, action binding and event handling. In some cases it would’ve been much nicer to just inline e.g. the button actions in normal code, somehow.
Then I watched the excellent video about Immediate Mode GUIs from Casey Muratori, read some of the posts on Molly Rocket’s forums and decided that I wanted an IMGUI too. But I didn’t like all of the features of a classic IMGUI, such as the manual ID management, stuffing all logic of widgets only in functions and delegating a lot of power to the widget, instead of something that looks upon it. Of course, there were alternatives, some were even proposed in the follow-up posts.
So the idea was to create a GUI that lives somewhere between immediate and retained modes. The API must be immediate mode, but widgets should be able to store any state behind the scenes without the user programmer explicitly saving it somewhere. I wanted to be able to define widgets as simple classes, by overriding a few methods and leaving the rest to a common base. This is how the prorotype of Hybrid was conceived. It turned out ot be a bit messy because of some rather weird design choices at the very beginning, but it worked really well for Deadlock and some toy apps, such as a draft of a shader editor:
It was pretty pleasant to work with, in general. Yet, some ID management quirks were still there, and well – it was just a prototype.
Some time after finishing Deadlock, I took a univ course in ‘Software Analysis and Engineering’. After complaining to the lecturer, that the Waterfall model is not the best thing around, I was asked to make a non-standard project for the course. Most other folks on my year got project assignments and worked in teams. My objective was to create … documentation for Hybrid.
But well, who creates user-docs for a prototype? So I took the opportunity and rewrote Hybrid from scratch. This time I needed it to support some retained mode constructs in addition to the immediate mode API.
This is the result. And it makes nice pictures too:
So what’s the future of Hybrid? As written in the TODO, I’ve got some things to implement still. Probably easy style and css-ish widget-class management is going to be the most important. Animation of arbitrary properties will also be a must-have if it’s to become a useful game GUI toolkit. On top of that, I’ve got some widget orders that I need to deliver, such as a color picker… I’m considering (s)RGB + XYZ + HSV + Spectrum – based picking :P
There also remain a few usability and stability issues, but that’s the reason it’s a Beta and not a final release.
Filed under: Uncategorized | Comments (8)
January 19, 2008
The image in my ‘HAI WORLD!’ post, just as the following one …
… was rendered with DShade, a 3d software rasterizer of mine.
DShade is mostly an experiment at abusing the D compiler, created a year ago as a final project for my “Graphics and Multimedia” course.
Thanks to some of D’s features, it can do simple software vertex and pixel shaders with on-the-fly recompilation and reloading. It also supports bilinear-filtered texturing, perspective-correct attribute interpolation, Sutherland-Hodgeman clipping, backface culling, z-buffering and alpha blending.
The demo app features a very simple loader for q3 bsp levels, it can also display ‘HME’ models, a custom text-based format exported from 3dsmax.
As far as the software shaders are concerned, these are actually parts of D modules, that get mixed into a wider scope, where they can be inlined in tight rasterization loops. Additionally, some metaprogramming tricks detect the inputs and outputs of the shaders in order to automatically extract and interpolate all required vertex parameters needed for the shader.
Each shader gets mixed into a Renderer instance and compiled into a dynamic library, which can be recompiled and reloaded when needed.
The user interface to DShade is fairly simple. ‘WSAD’ keys control camera movement; Holding the right mouse button and moving the mouse pointer allows camera rotation; The left button is used to bring up a context menu for objects. From this menu, shaders and textures can be changed, hierarchies between objects may be defined and simple movement can be specified. Hit space to bring up a menu for object creation.
While the program is running, try editing one of the files in the shaders/ directory. Saving it should trigger an automatic recompilation and reloading of it within DShade. The console window will contain error logs if something goes wrong.
Please note that DShade doesn’t make use of SSE or 3dNow or MMX, it may contain bugs and can bring very fast CPUs to their knees. If shaders don’t want to recompile, make sure you have DShade located in a path not containing spaces. Build/DMD/OPTLINK can get a bit lost with them.
Also note that DShade was conceived when DDL’s OMF backend wasn’t very stable. If I were to rewrite it, I’d definitely use DDL on Windows.
Hint: deleting the level from within the context menu will speed things up a lot.
You can download DShade along with its source code and play around (run Main.exe). Out of my pure laziness, the shader compilation and loading system only supports Windows.
P.S. Thanks to Walter Bright for allowing me to distribute a stripped-down version of DMD and OPTLINK with DShade :)
Filed under: Uncategorized | Comments (5)
