Life after D
Quote from tobega on March 28, 2021, 11:03 amQuote from Dave Voorhis on March 28, 2021, 12:27 amQuote from dandl on March 28, 2021, 12:00 amQuote from tobega on March 27, 2021, 1:50 pmQuote from dandl on March 27, 2021, 9:52 amM is a meta-programming language, for writing programs that read and write programs. Macros are a subset of M.
MJ is M for Java. I have just learned about the Java annotations processor, which seems to be a clunky subset of MJ. Java has nothing else.
I am highly confident that you cannot carry out the 4 tasks I outlined (and many others) with non-M "languages with desirable language features that compile (of which transpilation is a subset) to an object language target". If you are hostile to M/MJ, you will find workarounds and partial solutions, but I am confident that no matter what you propose, I can produce an example that M/MJ can solve and that what you propose cannot. If you are using any tools of any kind to generate or modify Java source code you are already acknowledging the need for MJ, but those are just the tip of the iceberg. Building serious software needs M.
I intend to follow up by finding others who are successfully using meta-programming and languages that show how it can be done.
I was rather surprised when the talk about "shorter-safer-higher" ended up in meta-programming. Shorter, sure, higher, I suppose, but safer? Most developers have a hard enough time producing straight first-order procedural code, never mind thinking about code that makes code that does something. So, sure, if you can produce a system that is usable, all power to you! (As a side note, the talk of the language you need vs the language you have made me think of Guy Steele's excellent classic talk on "growing a language" https://www.cs.virginia.edu/~evans/cs655/readings/steele.pdf or if you prefer to watch a video https://www.youtube.com/watch?v=_ahvzDzKdB0 )
<aside>As an anecdote, in about 2004 I built a system where the html page could be suitably annotated and then transformed by one XSLT template into another XSLT template that would be used to transform the XML data into a page with data. Hugely successful because the business people came in one week before going live with a new html page that we could just annotate and jack in. But the other developers found it ridiculously difficult so they ripped it out in favour of ASP.NET pages. They weren't even able to comprehend how XSLT worked, while the business people easily learned to produce XSLT templates (I held classes for both categories). The developers were too stuck in thoughts like "<html> is the command that begins an html document".</aside>
There is a lot of meta-programming in use today, much of it has been mentioned in threads here as Spring, Lombok, JPA and more. Spring used to be configured (and still can be) by xml files, but it is easier and better to use annotations as a virtual sort of code-transforming function (and Lombok and JPA do the same). Some of it is fairly useful, like an "@Post" annotation in Spring taking care of all the tedium of dealing with the http-protocol and just fills in the expected data into your method signature. Although if you just use an embedded Undertow server, you have a nice OO way of doing it instead, without the huge overhead of the Spring framework. But of course the codeshitters (love the term, Erwin!) find it much more interesting to produce frameworks for other codeshitters than to produce working software for real users.
Now these meta-programming constructs are just for consumption by the general programmer, it is not an easy thing to get into producing them. Spring has taken care to always provide ways to customize the behaviour when it doesn't quite fit your needs, but even that is not easy and is often severely underdocumented.
Your points are well made, and mesh nicely with my concept of the tool maker vs tool builder. In my experience tool makers are about 1 programmer in 10. 9 of 10 want to work 'concretely' with the end product using whatever tools they have and can understand and rely on. I belong to the 1 in 10 that prefer to think: what tool can I build to make this job much easier? Maybe you do too. [A few like Dave straddle the fence.]
But it behoves us tool-makes to put a complete, working, finished, reliable tool in the hands of the tool users, and you didn't do that. Your XSLT page was not a finished, reliable tool. Spring falls short if it's not documented. I wrote nearly 2000 pages of documentation for Powerflex, and it still falls short in places. [But I rely on it heavily for the bits I've forgotten.]
Veering off-topic here, but Spring is heavily documented. There are perhaps edge cases and work-in-progress bits, but the vast majority of Spring in daily use has a plethora of vendor-supplied documentation (https://spring.io), books (https://medium.com/javarevisited/10-best-spring-framework-books-for-java-developers-360284c37036), and online tutorials (https://www.baeldung.com/spring-tutorial).
That said, Spring is not magic, it's not a panacea, and there is plenty not to like about it, but lack of documentation isn't it.
For whatever weaknesses Java and its popular frameworks and libraries may have, weak documentation is rarely one of them.
True, for the normal expected uses the documentation is fairly to very good with plenty of tutorials. The problem here is rather that it is needed at all, i.e. it is not Java. Many things are not obvious what they do, perhaps especially when considering how different annotations interplay, but there are also gotchas in more normal use, e.g. that calling the method from within the same class bypasses the generated code.
For the case where you want to do something different than they anticipated, the documentation is usually not great. That's to be expected, of course, since you essentially have to understand enough about their implementation to override the pertinent parts, if it's even possible to do what you wanted.
Most of the time it would probably have been easier to do everything yourself with the aid of libraries of ordinary Java objects, then you would understand the process better to be able to deviate where needed, but Spring led you into a trap where you understand nothing beyond that their annotation magically solves a few predetermined cases that you might normally want. Frameworks make easy things trivial and difficult things impossible.
Yes, Spring is -- like many frameworks and other do-it-for-you mechanisms -- a "happy path" tool. If your requirements are on the happy path, it all slides together nicely per the documentation. If you step off the happy path, it all goes sour. Quickly.
I see a growing resistance to annotations because of this, particularly because they're essentially outside Java and can't be easily orchestrated by and in Java.
I'm inclined to join that resistance movement, to the point that if a tool isn't built from -- and emitting, if necessary -- plain old Java objects that I can use to do plain old Java things, sans annotations, then I don't want it even if it saves me time and effort here and there.
That's because I know from bitter experience that if it isn't purely plain old Java objects that I can use to do plain old Java things, I'm inevitably going to hit a wall with it -- probably right before a deadline or (worse) whilst I'm rushing to deliver a hotfix.
Quote from Dave Voorhis on March 28, 2021, 12:02 amQuote from dandl on March 27, 2021, 11:21 pmQuote from Dave Voorhis on March 27, 2021, 10:38 amQuote from dandl on March 27, 2021, 9:52 amM is a meta-programming language, for writing programs that read and write programs. Macros are a subset of M.
MJ is M for Java. I have just learned about the Java annotations processor, which seems to be a clunky subset of MJ. Java has nothing else.
I am highly confident that you cannot carry out the 4 tasks I outlined (and many others) with non-M "languages with desirable language features that compile (of which transpilation is a subset) to an object language target". If you are hostile to M/MJ, you will find workarounds and partial solutions, but I am confident that no matter what you propose, I can produce an example that M/MJ can solve and that what you propose cannot. If you are using any tools of any kind to generate or modify Java source code you are already acknowledging the need for MJ, but those are just the tip of the iceberg. Building serious software needs M.
I intend to follow up by finding others who are successfully using meta-programming and languages that show how it can be done.
I don't think M exists yet, let alone MJ, so it's a bit difficult to evaluate it (except from your arm-waving about it, I guess.)
Using tools to generate (not modify) Java source code is not a need for MJ, but simply a recognition that pre-existing static structures in external definitions -- like XML DTDs or SQL schemas or RESTful API specifications -- may benefit from having static Java class or record analogues, and creating them can be automated.
That isn't a need for a distinct language of any kind; that's simply integration of pre-existing external data definitions into Java. You typically aren't writing the XML DTDs or SQL schemas or RESTful API specifications because they exist already (and exposing Java as a RESTful API is a good and proper use of Java annotations.)
You're merely running this Java program that's pointed to them, and Java code appears over there as if you'd written it by hand. You're still working entirely -- and desirably -- in a Java domain.
Any sufficiently complicated Java program contains an ad hoc, informally-specified, bug-ridden, slow implementation of parts of MJ.
I don't think anything contains any kind of implementation of MJ because it doesn't exist, even as a specification. Thus far, it's not sufficiently defined to be considered more than a vague idea.
It's very precisely specified: meta-programming (programs that read and write programs as data) within the Java ecosystem. The intention within this thread was to restrict it to inputs and outputs of textual form. The claim is that doing so would be more productive than tweaking Java or creating a new language. It remains to be seen whether any particular language could or should be defined to fall within those parameters, possibly as extensions to the Java language, depending largely on what problems it sets out to solve.
And I don't think you ever did address the question of whether those 4 tasks could be addressed with a non-MJ language, and if so how.
Re "successfully using meta-programming and languages that show how it can be done," take a look at Nim (it even has macros!) and Haxe (with macros! https://haxe.org/manual/macro.html)
Though their macros are not text-replacement macros.
Yes, those are two languages I would look to for inspiration. They do indeed have macros, of the kind I had in mind even if you didn't think so.
Now take another look at your Java code generation and tell me if you could code that inline with macros like Nim or Haxe.
My own Java code generation -- and others that I use -- are based on pre-existing XML DTD, SQL schemas, resultsets for specified SQL queries, and RESTful API specs like WSDL. It's pre-existing standard-format specification in; Java out. As such, whilst Nim and Haxe are interesting in and of themselves, they don't serve a purpose for the work I do.
If a hypothetical M or MJ will solve problems in your domain, excellent.
But based on the idea of it, I don't see it solving problems in mine. Though I'm happy to defer final evaluation to the point at which it exists.
Again, there is enough detail in the specs of Nim or Haxe for you to express an opinion as to whether your code generation could be done that way. I didn't ask whether you would like to do it that way, but others might.
The only constraint I would place is that the developer should easily be able to see the expanded code in a readable format, preferably in the editor. Debugging macros without knowing exactly what the compiler turned it into is too hard.
Yes, though note that whilst this is helpful to the professional programmer, it's unlikely to be helpful to the semi-technical developer that I understand you're targeting. If he's using M to generate the code he doesn't want to -- or can't -- write, I'm not sure what will be gained by showing him the code he wouldn't or couldn't write.
Better, I think, that the language semantics be self-standing and regulated, so that the compiler can't be (mis)led into generating code that demands viewing in order to make sense of it (except if you're the compiler author, of course.)
That's a paper tiger. The specific tasks I proposed are only of interest to serious professional programmers, and IMO they would very much value the ability to look into the generated code, particularly while developing libraries or tools for others to use. Business domain experts with limited programming tools need different aids, which elide inessential detail and present results in a business/domain context. MJ can do that too.
For whatever weaknesses Java and its popular frameworks and libraries may have, weak documentation is rarely one of them.
That's a tautological argument. Frameworks with weak documentation are not popular.
AOP was intended to be a general way to add cross-cutting concerns, but on the C2 wiki even the supporters of it concede that there usually are better object-oriented ways of doing it. It doesn't look too difficult to do, but there seem to be some inherent problems in how they get applied and composed. In functional-land I think they would use Monads instead which are composable language constructs and so much better. And I suppose the ML module system is also an excellent programming construct for this kind of thing (but I admit I have still a very hazy idea of how that works).
I know AOP but this is wrong. Case in point: I want a small library of tracing/logging functions I can add to my C# programs, with the following features:
- It produces nicely formatted text output including options for date, time, argument values, etc
- It uses the minimum amount of space on the output device (collapse/merge similar messages)
- It can send output to the console, window, text file, debugger or stream
- It works on any platform: Windows console/UI/DLL/service, Unity, Unix, etc
- Trace calls can be enabled/disabled at runtime by program or configuration
- Disabled trace calls are 'free' (little or no CPU impact)
- Trace calls can be excluded from a production build
You can't do that with objects, but you can do it with C++ templates and macros.
That's standard logging; ILogger and friends in .NET, and a number of tools both inbuilt (java.util.logging) and popular 3rd party like log4j2 in Java. I'm even a minor contributor to that world with my rather specialist (logs by default to a SQLite database) https://github.com/DaveVoorhis/dbLogger
All you've described and more is typical of modern logging frameworks. The only missing (if we can call it that) bits are "Disabled trace calls are 'free'" and (this one baffles me) "Trace calls can be excluded from a production build."
Not really. I've used those products, and they're certainly better than they used to be, but there are still gaps. From what I can see a lot of the work has been pushed onto the build system, so I guess there are chunks of conditional code, with a degree of duplication, but it's too hard to go look. What about Android? JNI?
[BTW I'll add assert functionality, with much the same set of requirements.]
First, disabled trace calls are not free. But they're very low cost, usually costing as little as almost nothing for the invocation of stub calls via a 'NoLogging' or similar implementation, plus the config load or activation/deactivation time, which is always present with a logging framework because you want to be able to turn it on/off at least at launch, if not at runtime, for production debugging.
Free means: no measurable performance impact, as against no theoretical impact. Yes, free is almost always achievable in C/C++. The closest you get in Java is a method call with multiple parameters that returns immediately. At least now you can use lambdas to avoid the worst of it, but MJ could do better.
The second one, "Trace calls can be excluded from a production build": Logging disabled or turned down to some minimal level during normal production is typical, along with the ability to turn logging on or up to any level, but being not-compiled-into-the-code excluded from enabling logging in production systems is oh-my-god levels of bad.
No, that's wrong. A developer/research/pre-alpha can and should have a level of instrumentation that is completely unacceptable in a finished product. At least part of the instrumentation code needs to be removed from the final production build but retained for future use.
The important thing is that features of this kind are routinely used in building major software products in C/C++, and if comparable features were provided by MJ, there are people who would use them but who now resort to various workarounds.
Quote from dandl on March 29, 2021, 1:45 amFor whatever weaknesses Java and its popular frameworks and libraries may have, weak documentation is rarely one of them.
That's a tautological argument. Frameworks with weak documentation are not popular.
Indeed. But I regularly encounter badly-documented but well-used tools in the C# world. This is the sort of thing I see again, and again, and again: https://joshclose.github.io/CsvHelper/api/
It's been showing that for several months now and CsvHelper is quite popular.
Contrast that with the Java equivalent: https://commons.apache.org/proper/commons-csv/user-guide.html
AOP was intended to be a general way to add cross-cutting concerns, but on the C2 wiki even the supporters of it concede that there usually are better object-oriented ways of doing it. It doesn't look too difficult to do, but there seem to be some inherent problems in how they get applied and composed. In functional-land I think they would use Monads instead which are composable language constructs and so much better. And I suppose the ML module system is also an excellent programming construct for this kind of thing (but I admit I have still a very hazy idea of how that works).
I know AOP but this is wrong. Case in point: I want a small library of tracing/logging functions I can add to my C# programs, with the following features:
- It produces nicely formatted text output including options for date, time, argument values, etc
- It uses the minimum amount of space on the output device (collapse/merge similar messages)
- It can send output to the console, window, text file, debugger or stream
- It works on any platform: Windows console/UI/DLL/service, Unity, Unix, etc
- Trace calls can be enabled/disabled at runtime by program or configuration
- Disabled trace calls are 'free' (little or no CPU impact)
- Trace calls can be excluded from a production build
You can't do that with objects, but you can do it with C++ templates and macros.
That's standard logging; ILogger and friends in .NET, and a number of tools both inbuilt (java.util.logging) and popular 3rd party like log4j2 in Java. I'm even a minor contributor to that world with my rather specialist (logs by default to a SQLite database) https://github.com/DaveVoorhis/dbLogger
All you've described and more is typical of modern logging frameworks. The only missing (if we can call it that) bits are "Disabled trace calls are 'free'" and (this one baffles me) "Trace calls can be excluded from a production build."
Not really. I've used those products, and they're certainly better than they used to be, but there are still gaps. From what I can see a lot of the work has been pushed onto the build system, so I guess there are chunks of conditional code, with a degree of duplication, but it's too hard to go look. What about Android? JNI?
[BTW I'll add assert functionality, with much the same set of requirements.]
First, disabled trace calls are not free. But they're very low cost, usually costing as little as almost nothing for the invocation of stub calls via a 'NoLogging' or similar implementation, plus the config load or activation/deactivation time, which is always present with a logging framework because you want to be able to turn it on/off at least at launch, if not at runtime, for production debugging.
Free means: no measurable performance impact, as against no theoretical impact. Yes, free is almost always achievable in C/C++. The closest you get in Java is a method call with multiple parameters that returns immediately. At least now you can use lambdas to avoid the worst of it, but MJ could do better.
The second one, "Trace calls can be excluded from a production build": Logging disabled or turned down to some minimal level during normal production is typical, along with the ability to turn logging on or up to any level, but being not-compiled-into-the-code excluded from enabling logging in production systems is oh-my-god levels of bad.
No, that's wrong. A developer/research/pre-alpha can and should have a level of instrumentation that is completely unacceptable in a finished product. At least part of the instrumentation code needs to be removed from the final production build but retained for future use.
The important thing is that features of this kind are routinely used in building major software products in C/C++, and if comparable features were provided by MJ, there are people who would use them but who now resort to various workarounds.
You appear to be suggesting M or MJ or whatever for conditional compilation in-code. That's practically the antithesis of certain server-side work I'm paid to do, and your approach completely at odds with current thinking in that field -- indeed, it not only has negative technical implications; for some systems I've worked on, it even has potential regulatory impact.
Therefore, I'm going to bow out of this discussion here. If what you're suggesting for M/MJ suits your work, excellent. From what I've seen so far, it doesn't suit mine.
I just came across a language project, Wyvern, that is trying to go "shorter, safer, higher" by lifting programming more towards the design level. They also have the idea of type-specific languages, which seem to be a bit like macros, but is restricted to producing one value of a particular datatype, according to a language/notation designed to be more appropriate for that type, which is then converted in the compilation stage to more basic structures of the host language.