The future of TTM
Quote from Dave Voorhis on August 2, 2021, 3:01 pmQuote from dandl on August 2, 2021, 2:15 pmQuote from Dave Voorhis on August 2, 2021, 7:27 amQuote from dandl on August 2, 2021, 12:10 amQuote from Dave Voorhis on August 1, 2021, 8:35 amQuote from Darren Duncan on August 1, 2021, 5:38 am...
What are thoughts on this?
Build it and see what you get.
As an aside, and not directed at you (and directed at myself as much as anyone...)
Time to stop cackling and lay an egg.
We do a lot of cackling and not much egg laying here. If we were a bunch of hens, we'd have been branded useless and tossed in the soup pot long ago.
Write some code and get it on github or wherever and try to use it to do real things. Whether it starts with the relational model and a search for applications, or starts with solving a problem or requirements and uses the relational model as a solution, or something else, doesn't really matter.
What matters is that if there's to be any future in TTM ideas, it will be in the form of written and working and useable code, not endless cackling about code-that-might-be.
Afer some thought, I have to disagree. Writing code is easy, the hard things are (a) knowing what code to write and (b) getting others to use it.
You have produced a high quality implementation of a TTM/D. It has a strong usage case as a teaching aide, backed up by D&D textbooks. But does anyone else use it?
Erwin's SIRA_PRISE is an impressive piece of coding, but does anyone use it? My Andl and two subsequent related products are a major investment in time and effort, but does anyone use it? This is working and usable code but the answer is no, not even me. It doesn't solve anyone's problem.
So, IMO what matters is that if there's to be any future in TTM ideas, it will be in the form of code that solves someone's problem. Writing the code is easy, it's finding the right problem that's hard. And that's what I've been trying to do.
I think you're trying to solve someone else's problem.
Write code to solve your own. Then you're guaranteed at least one happy user.
Any more than that is a bonus, but I suspect you'll be much more likely to get more users by solving your own real problems than trying to find someone else's problems to solve.
That may work for you, it doesn't for me. I just don't have any problems that need this kind of tech to solve. If I did, I would do just enough to scrape by, never enough to be good enough for someone else to use.
We have just had the most successful unicorn ever. Afterpay founded in 2014 and just sold for $A39 billion. The founders of Afterpay did not write software so that they personally could 'buy now pay later', or to use in their own retail business. They wrote software to solve someone else's problem: retailers who wanted to make online sales with instant credit and minimal regulation.
Conversely, two of the biggest wins in terms of scoring everybody-uses-it market share levels are Slack and git, both of which were developers scratching their own -- or at least their development team's -- itches.
Two excellent examples of building software to solve a known problem for a known customer: a game development team and Linux developers/contributors respectively. Find the customer problem, write the code, profit!
Some months ago, you were looking for something to do desktop data dicing and slicing. That, if I recall correctly, led you to work on extending KNIME. But at the time it sounded to me like a perfect opportunity to dust off Andl and reshape it to solving the problem -- and the general category of problems like it -- with or without KNIME.
For me, the work I did was to learn some Java and look for problems I could solve in the data analytics community. I didn't find (or solve) a customer problem, but it was entertaining, to a point. And I have a good grasp of reasons why this is not a friendly place for TTM ideas.
Then, you talked about a colleague using Excel to awkwardly track something, so you wanted automatic CRUD-UI-generation. Not sure what happened with that.
That's the problem that Knack and Google Sheets set out to solve, but in the cloud. I worked on it for a while to learn JS and React, which convinced me those are tools I don't want to use. In fact, it turns out to be really simple problem at the Computer Science level, and a really tough problem at the UI level. That's a problem worth solving, but how?
To me, rather than "I just don't have any problems that need this kind of tech to solve", it sounds like you have exactly the sort of "problems that need this kind of tech to solve" popping up under your nose and shouting "me me me me!"
And indeed the Not Another Excel problem (as I call it) is the only one I'm personally interested to pursue. It falls into 2 parts: UI and database. The database part looks exactly like what this thread has been about: in-process database with relational features. Easy, just pick a database and go for it. The UI part is still the tough nut, and I currently have no candidates I'd be happy to use.
Otherwise, trying to land big fish in the general problem-solving world is either standard consulting -- which is almost 100% off-the-shelf platforms and tools -- or new products that demand lottery-win levels of luck and fortuity, along with bet-your-life-savings-and-all-your-assets levels of marketing and promotion.
Indeed, for every unicorn that succeeds wildly, there are a million or so expensive attempts that don't.
My formula is always: find a customer with a problem (or need), solve the problem, get paid, rinse and repeat. That's business in a nutshell.
That's (software) engineering consulting in a nutshell, which uses standard toolchains and good marketing and a strong reputation to succeed.
Entrepreneurial innovation based on a new product is rather different, particularly as you may not even be solving a problem as much as creating a new problem space in which new problems can be solved.
Again, I can't agree. The standard consultant model depends on acquiring the customer and then solving lots of problems for the same customer. The startup is the opposite: find lots of customers with the same problem. Bring a new problem to a customer is expensive -- you have to educate the customer and what you each them may just help a competitor. Solving a problem the customer is already desperate to solve is far cheaper -- they sell themselves.
Linus and team already deeply understood the problem (of distributed version control), it was the novel solution that made it succeed. Find a customer with an urgent, compelling problem and offer them a novel solution that is faster/cheaper/better and do find many more so you can do it at scale, that's what makes a successful startup.
Quote from Erwin on August 3, 2021, 9:51 amQuote from dandl on August 2, 2021, 12:10 amAfer some thought, I have to disagree. Writing code is easy, the hard things are (a) knowing what code to write and (b) getting others to use it.
You have produced a high quality implementation of a TTM/D. It has a strong usage case as a teaching aide, backed up by D&D textbooks. But does anyone else use it?
Erwin's SIRA_PRISE is an impressive piece of coding, but does anyone use it? My Andl and two subsequent related products are a major investment in time and effort, but does anyone use it? This is working and usable code but the answer is no, not even me. It doesn't solve anyone's problem.
So, IMO what matters is that if there's to be any future in TTM ideas, it will be in the form of code that solves someone's problem. Writing the code is easy, it's finding the right problem that's hard. And that's what I've been trying to do.
Erwin's SIRA_PRISE does solve a problem that everyone is having : the fact that they need programmers/coders to get the business rules enforced (and the miserable quality and reliability of the data that inevitably ensues). But for one reason or other, it doesn't seem to be seen as a real problem by those experiencing it. Well, their loss not mine.
I had my personal reasons for doing that project and that was mainly I wanted to know purely for myself how far I could get. I have done so and I know it.
It's an impressive piece of work and I understand perfectly the motivation for doing it. Andl is in the same camp.
The key problem here I think is that people don't like to switch languages. They have a problem, they see a solution, but the price is too high.
Perhaps if it could be reformulated into a library callable from Java it might get more traction?
Quote from dandl on August 3, 2021, 1:53 pmQuote from Erwin on August 3, 2021, 9:51 amQuote from dandl on August 2, 2021, 12:10 amAfer some thought, I have to disagree. Writing code is easy, the hard things are (a) knowing what code to write and (b) getting others to use it.
You have produced a high quality implementation of a TTM/D. It has a strong usage case as a teaching aide, backed up by D&D textbooks. But does anyone else use it?
Erwin's SIRA_PRISE is an impressive piece of coding, but does anyone use it? My Andl and two subsequent related products are a major investment in time and effort, but does anyone use it? This is working and usable code but the answer is no, not even me. It doesn't solve anyone's problem.
So, IMO what matters is that if there's to be any future in TTM ideas, it will be in the form of code that solves someone's problem. Writing the code is easy, it's finding the right problem that's hard. And that's what I've been trying to do.
Erwin's SIRA_PRISE does solve a problem that everyone is having : the fact that they need programmers/coders to get the business rules enforced (and the miserable quality and reliability of the data that inevitably ensues). But for one reason or other, it doesn't seem to be seen as a real problem by those experiencing it. Well, their loss not mine.
I had my personal reasons for doing that project and that was mainly I wanted to know purely for myself how far I could get. I have done so and I know it.
It's an impressive piece of work and I understand perfectly the motivation for doing it. Andl is in the same camp.
The key problem here I think is that people don't like to switch languages. They have a problem, they see a solution, but the price is too high.
Perhaps if it could be reformulated into a library callable from Java it might get more traction?
Quote from dandl on August 3, 2021, 1:53 pmQuote from Erwin on August 3, 2021, 9:51 amQuote from dandl on August 2, 2021, 12:10 amAfer some thought, I have to disagree. Writing code is easy, the hard things are (a) knowing what code to write and (b) getting others to use it.
You have produced a high quality implementation of a TTM/D. It has a strong usage case as a teaching aide, backed up by D&D textbooks. But does anyone else use it?
Erwin's SIRA_PRISE is an impressive piece of coding, but does anyone use it? My Andl and two subsequent related products are a major investment in time and effort, but does anyone use it? This is working and usable code but the answer is no, not even me. It doesn't solve anyone's problem.
So, IMO what matters is that if there's to be any future in TTM ideas, it will be in the form of code that solves someone's problem. Writing the code is easy, it's finding the right problem that's hard. And that's what I've been trying to do.
Erwin's SIRA_PRISE does solve a problem that everyone is having : the fact that they need programmers/coders to get the business rules enforced (and the miserable quality and reliability of the data that inevitably ensues). But for one reason or other, it doesn't seem to be seen as a real problem by those experiencing it. Well, their loss not mine.
I had my personal reasons for doing that project and that was mainly I wanted to know purely for myself how far I could get. I have done so and I know it.
It's an impressive piece of work and I understand perfectly the motivation for doing it. Andl is in the same camp.
The key problem here I think is that people don't like to switch languages. They have a problem, they see a solution, but the price is too high.
Perhaps if it could be reformulated into a library callable from Java it might get more traction?
There ***is*** a "library callable from Java". It gets you connections, start and end transactions, issue queries, issue updates, translate query results to sets of Java objects not entirely unlike your average ORM, change state in such objects and issue db updates reflecting those state changes.
Ii really don't know what else is left to wish for in a db connectivity library.
https://shark.armchair.mb.ca/~erwin/doc0105/public/index.html
Erwin's SIRA_PRISE is an impressive piece of coding, but does anyone use it? My Andl and two subsequent related products are a major investment in time and effort, but does anyone use it? This is working and usable code but the answer is no, not even me. It doesn't solve anyone's problem.
So, IMO what matters is that if there's to be any future in TTM ideas, it will be in the form of code that solves someone's problem. Writing the code is easy, it's finding the right problem that's hard. And that's what I've been trying to do.
Erwin's SIRA_PRISE does solve a problem that everyone is having : the fact that they need programmers/coders to get the business rules enforced (and the miserable quality and reliability of the data that inevitably ensues). But for one reason or other, it doesn't seem to be seen as a real problem by those experiencing it. Well, their loss not mine.
I had my personal reasons for doing that project and that was mainly I wanted to know purely for myself how far I could get. I have done so and I know it.
It's an impressive piece of work and I understand perfectly the motivation for doing it. Andl is in the same camp.
The key problem here I think is that people don't like to switch languages. They have a problem, they see a solution, but the price is too high.
Perhaps if it could be reformulated into a library callable from Java it might get more traction?
There ***is*** a "library callable from Java". It gets you connections, start and end transactions, issue queries, issue updates, translate query results to sets of Java objects not entirely unlike your average ORM, change state in such objects and issue db updates reflecting those state changes.
Ii really don't know what else is left to wish for in a db connectivity library.
https://shark.armchair.mb.ca/~erwin/doc0105/public/index.html
I didn't know that. Can you point me to somewhere on the web site that explains how that works and how I would use it in an application? With examples?
I have always assumed that it's a scripting language, with a server and web client lurking somewhere in the background. Mostly I've referred to the command language page, and dipped into the others. I assumed the Javadoc was for internal reference only. It seems I was mistaken.
[I could make some suggestions about how to make it more approachable:
- Fix the front page of the Web site to better explain what it is, what it can do, who it's for and how it can be used
- Get a a dedicated domain for the web site
- Host it on GitHub not SourceForge
- Show me the code: samples of what it can do with source code in script and Java.
But I don't know how they would be taken.]
So, I have a problem. I have a database dump that exists as 20 CSV files. I would like to load it into a new database so that I can view, report and maintain the data. How much of that will SIRA_PRISE do, and how?
Quote from dandl on August 4, 2021, 1:52 amI didn't know that. Can you point me to somewhere on the web site that explains how that works and how I would use it in an application? With examples?
I have always assumed that it's a scripting language, with a server and web client lurking somewhere in the background. Mostly I've referred to the command language page, and dipped into the others. I assumed the Javadoc was for internal reference only. It seems I was mistaken.
[I could make some suggestions about how to make it more approachable:
- Fix the front page of the Web site to better explain what it is, what it can do, who it's for and how it can be used
- Get a a dedicated domain for the web site
- Host it on GitHub not SourceForge
- Show me the code: samples of what it can do with source code in script and Java.
But I don't know how they would be taken.]
So, I have a problem. I have a database dump that exists as 20 CSV files. I would like to load it into a new database so that I can view, report and maintain the data. How much of that will SIRA_PRISE do, and how?
Quote from dandl on August 4, 2021, 1:52 am
So, I have a problem. I have a database dump that exists as 20 CSV files. I would like to load it into a new database so that I can view, report and maintain the data. How much of that will SIRA_PRISE do, and how?
2nd answer.
- Design your SIRA_PRISE database (which you are going to have to do regardless of whether you want to "load" it or not)
- Use the DBObjectGenerator to generate 20 java DBObject classes reflecting your (presumably) 20 relvars.
- Manually complete the 20 java classes to eliminate all the flaws the generator still leaves in the generated source (despite the fact that the generator is part of 1.5, the generator itself is 1.0, or maybe even just 0.1)
- Write your 20 java clients to read and parse the CSV at hand, create the appropriate DBObject from the data obtained from the parsing, and issue the insert. If the load volume gets into the higher numbers, tune the commit rate and the number of tuples per insert.
- You have not mentioned constraints. So i did not consider constraints. Constraints cannot be checked on CSV's. Resolving "constraint violations" after the loads heve been done becomes part of the problem of introducing the constraints into the design. (The user who ***does get*** the SIRA_PRISE mindset would firstly ask himself why he'd want to "load" data that is invalid - which is exactly what the 'C' in 'ACID' is all about - invalid data won't be accepted - which is exactly the reason why there's a proposal to just 'drop ACID' : because there are people who believe that it's the wrong mindset and the wrong order of operations to load data first and solve the semantic problems later.)
I'd also like to point out that points 2. and 3. have a lasting effect on productivity because the result of those steps can subsequently be used by any program approaching the concerned relvars. So in a sense (economically speaking, mostly) it is also a part of the "database design" effort, not the "database load" effort.
Quote from Erwin on August 4, 2021, 3:29 pmQuote from dandl on August 4, 2021, 1:52 amI didn't know that. Can you point me to somewhere on the web site that explains how that works and how I would use it in an application? With examples?
I have always assumed that it's a scripting language, with a server and web client lurking somewhere in the background. Mostly I've referred to the command language page, and dipped into the others. I assumed the Javadoc was for internal reference only. It seems I was mistaken.
[I could make some suggestions about how to make it more approachable:
- Fix the front page of the Web site to better explain what it is, what it can do, who it's for and how it can be used
- Get a a dedicated domain for the web site
- Host it on GitHub not SourceForge
- Show me the code: samples of what it can do with source code in script and Java.
But I don't know how they would be taken.]
So, I have a problem. I have a database dump that exists as 20 CSV files. I would like to load it into a new database so that I can view, report and maintain the data. How much of that will SIRA_PRISE do, and how?
Curiously, yesterday I was looking for the technical reference manual of the berkeley db system because I had the exact same questions regarding their system as you about mine. How does it work ? How do I use it ? What are the API's I can call ? It appears not to exist. Unless of course there's a fundamental incompatibility between the Erwin Smout way of searching and the perception that Oracle has of how interested users are going to search. Anyway, that's ***ORACLE***. Pls don't come and tell me they don't have the means to write a decent manual and publish it (and make it findable).
There is an "architectural overview" doc at https://shark.armchair.mb.ca/~erwin/ArchitecturalOverview_0105.pdf . It literally mentions "Your java app" (a.o.) Granted I could have added some explanations but does "Your java app" really need more explanation than what the words already say ? Really ?The very first line on the start page says literally "What is SIRA_PRISE", immediately followed by what should be looking like a clickable link. People who do not get that by clicking that link they might land on a page that has answers to that question, are not part of the intended audience.People who assess the value of a software product by where it's hosted, are not part of the intended audience either, btw.Demo applications were included. The admin webclient is itself also such an application and it's also included. All of those were publicly available. Open source so "publicly available" included the sources.You see that section on the start page "the SIRA_PRISE language" ? It has three subsections. That's because when there was only the first subsection, I got comments "that's not how you should do it". So I tried another approach and that became the second subsection. I still got comments of the ilk "that's not really what I'm looking for". So I tried to come up with yet another approach and that became subsection 3. And when that was ready for publication I decided "this is it". No more subsection 4 because some dude somewhere wants to see things explained his personal private way. They've got to do some effort too.There ***IS*** a script processor but it's pure sequential DML. No IF, no WHILE, no local variables. Turing complete you do that in java, or any other language provided the connectivity library gets written for that language. I have lost count of the number of times I have added that disclaimer in any discussion of the TTM projects offering "computationally complete language".
The CSV : write a java client to read and parse the CSV, issue insert statements in whatever relation cardinality is appropriate for your use case. An import module to convert "popular formats" (ods and such) to relations would have helped, but each time I intended to start building it I got visions of, well, never mind actually. So it never got built. That doesn't mean it's not doable.Something "tutorial-like" of "how to use the client library" is indeed missing. Technical reference vs. user guide ... Both are needed. But hey, you show me the unpaid-for software product that has both on a decent level of quality/usability ...
No-one is doubting the quality of the product, the question is entirely about how accessible it is. Can a mildly curious visitor figure out enough to write a Java app with it?
Quote from Erwin on August 4, 2021, 8:23 pmQuote from dandl on August 4, 2021, 1:52 amSo, I have a problem. I have a database dump that exists as 20 CSV files. I would like to load it into a new database so that I can view, report and maintain the data. How much of that will SIRA_PRISE do, and how?
2nd answer.
- Design your SIRA_PRISE database (which you are going to have to do regardless of whether you want to "load" it or not)
- Use the DBObjectGenerator to generate 20 java DBObject classes reflecting your (presumably) 20 relvars.
- Manually complete the 20 java classes to eliminate all the flaws the generator still leaves in the generated source (despite the fact that the generator is part of 1.5, the generator itself is 1.0, or maybe even just 0.1)
- Write your 20 java clients to read and parse the CSV at hand, create the appropriate DBObject from the data obtained from the parsing, and issue the insert. If the load volume gets into the higher numbers, tune the commit rate and the number of tuples per insert.
- You have not mentioned constraints. So i did not consider constraints. Constraints cannot be checked on CSV's. Resolving "constraint violations" after the loads heve been done becomes part of the problem of introducing the constraints into the design. (The user who ***does get*** the SIRA_PRISE mindset would firstly ask himself why he'd want to "load" data that is invalid - which is exactly what the 'C' in 'ACID' is all about - invalid data won't be accepted - which is exactly the reason why there's a proposal to just 'drop ACID' : because there are people who believe that it's the wrong mindset and the wrong order of operations to load data first and solve the semantic problems later.)
I'd also like to point out that points 2. and 3. have a lasting effect on productivity because the result of those steps can subsequently be used by any program approaching the concerned relvars. So in a sense (economically speaking, mostly) it is also a part of the "database design" effort, not the "database load" effort.
There are tools readily available to turn XLS/CSV files into Java objects (eg PIO). The question is how go about building a SIRA_PRISE app with the populated data model they represent, adding constraints as needed. I literally wouldn't know where to start.
Quote from dandl on August 2, 2021, 12:10 amWriting code is easy, the hard things are (a) knowing what code to write and (b) getting others to use it.
I was intrigued by this statement, and there doesn't seem to have been the response to it that I would have expected.
I presume you mean that you are a skilled programmer in one or more languages, so the hard part is (a) figuring out the architecture and algorithms to be expressed by the code, and (b) determining the sort of product to be produced by the architecture and algorithms. (b) should be solved before (a).
If so, I'd go along with you. If not, please ignore the rest of this post and post a correction for me.
My interest stems from the fact that I got into computing by having to solve problems for which it turned out I had to use a computer, i.e. learn to program and write the software to solve the problem. I then professionally got into Operation Research (= use scientific methods to solve management problems) which meant I did this all the time, since the problems typically had solutions solved by running software on computers.
I mention this because for me, your (a) and (b) have always been the essence of programming; whereas I'm not sure whether that is true for yourself.
I've also found that many commercial programmers seem more interested in code writing than (a) and (b). Their concerns are typically more about the nature of the programming language they're using, and the code's physical performance. These concerns are important of course, and essential to a good product. I'm not deriding them.
I suppose it's just that I've learned a different attitude.
Quote from dandl on August 5, 2021, 3:50 amIt's not my intention to debate this. I provided some impressions and suggestions that I thought might be helpful. Obviously not.I went round in circles looking for that 'start page', until I realised the 'Home' link is broken. You might want to fix that.As I said, I knew abut the scripting language, I just didn't know about the Java API and I still don't know how to use it.No-one is doubting the quality of the product, the question is entirely about how accessible it is. Can a mildly curious visitor figure out enough to write a Java app with it?
I've added stuff to the https://shark.armchair.mb.ca/~erwin/scriptprocessor_0105.html page. While doing so, I noted that the page already did say as its first sentence : "Two principal uses can be made of the provided client package : it can be used to communicate with a SIRA_PRISE server from within a java program, or it can be used to do "scripting" ". I've also addressed the broken home links.