Discussion:
O_TEXT for PDOS/386
(too old to reply)
Paul Edwards
2024-02-20 06:51:15 UTC
Permalink
Hi.

Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.

PDOS/386 (available at http://pdos.org) is able
to run a hello world Linux ELF binary, and recent
developments mean that I should be able to run
more than just a hello world soon.

But even with those recent developments, I am
still missing the same thing that Cygwin was
missing, and added O_TEXT to solve.

I'd like to add an O_TEXT flag to Linux. It would
be good if Linux officially reserved a flag for
this use, but in the absence of that, I'm happy
to create my own.

I think Linux adds flags going up, so I would add
any flags I want going down.

So I'm thinking of making O_TEXT 0x80000000

Another possibility would be 0x40000000 to avoid
being seen as a negative number.

This is different from Cygwin, which uses 0x20000.

I would be happy to use the Cygwin value, but I
think Linux may have already encroached that.

And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.

Any suggestions?

Thanks. Paul.
Richard Kettlewell
2024-02-20 09:53:25 UTC
Permalink
Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
quibbled) that the file is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
PDOS/386 (available at http://pdos.org) is able to run a hello world
Linux ELF binary, and recent developments mean that I should be able
to run more than just a hello world soon.
But even with those recent developments, I am still missing the same
thing that Cygwin was missing, and added O_TEXT to solve.
IMO Cygwin got this one wrong. The text/binary file distinction should
be managed in the C library or higher. Applications should either use
the appropriate arguments to fopen(), or manage any translation required
themselves.
I'd like to add an O_TEXT flag to Linux. It would be good if Linux
officially reserved a flag for this use, but in the absence of that,
I'm happy to create my own.
You’d need to ask on the linux-kernel mailing list, I think. But I
wouldn’t predict a very positive response.
--
https://www.greenend.org.uk/rjk/
Paul Edwards
2024-02-20 10:12:00 UTC
Permalink
Post by Richard Kettlewell
Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
quibbled) that the file is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
PDOS/386 (available at http://pdos.org) is able to run a hello world
Linux ELF binary, and recent developments mean that I should be able
to run more than just a hello world soon.
But even with those recent developments, I am still missing the same
thing that Cygwin was missing, and added O_TEXT to solve.
IMO Cygwin got this one wrong. The text/binary file distinction should
be managed in the C library or higher. Applications should either use
the appropriate arguments to fopen(), or manage any translation required
themselves.
Hi Richard. Thanks for your reply.

The C library that I use for Linux is a Linux port
of PDPCLIB, and is designed for Linux, and thus, no
translation will be done.

It is only when I run that Linux ELF binary on
PDOS/386 (mini Win32 clone) that I want translation
done. And even that would be a PDOS/386 configuration
thing. Maybe I would like to run a NL-only system.

So it can't easily (the non-Linux environment would
need to be detected, and assumptions made) be done in
the (Linux) C library, and it wouldn't be appropriate
either.
Post by Richard Kettlewell
I'd like to add an O_TEXT flag to Linux. It would be good if Linux
officially reserved a flag for this use, but in the absence of that,
I'm happy to create my own.
You’d need to ask on the linux-kernel mailing list, I think. But I
wouldn’t predict a very positive response.
That's what I'm predicting too. So can you suggest
a flag? I think Linux is consuming flags going up,
so I could go down for anything I need, and have
0x80000000 mean "O_TEXT".

But that makes it a negative number, which may not
be nice, so I could make it 0x40000000.

Another possibility would be to commandeer some
existing unimportant flag. I think there might be
one for "access time"? Since I don't care whether
or not access time is updated, if access time is
requested, I could make it mean "text".

Or some combination of flags.

Any suggestions within the constraints of the Linux
people having different goals to me?

Thanks. Paul.
Richard Kettlewell
2024-02-20 19:11:06 UTC
Permalink
Post by Paul Edwards
Post by Richard Kettlewell
I'd like to add an O_TEXT flag to Linux. It would be good if Linux
officially reserved a flag for this use, but in the absence of that,
I'm happy to create my own.
You’d need to ask on the linux-kernel mailing list, I think. But I
wouldn’t predict a very positive response.
That's what I'm predicting too. So can you suggest
a flag? I think Linux is consuming flags going up,
so I could go down for anything I need, and have
0x80000000 mean "O_TEXT".
No, I can’t suggest a flag. I have no more control over the Linux ABI
than you do. That’s why I pointed you to linux-kernel.
Post by Paul Edwards
Any suggestions within the constraints of the Linux people having
different goals to me?
I think you should rethink your goals and/or your approach to meeting
them. open/read/write is the wrong place to translate between line
ending conventions.
--
https://www.greenend.org.uk/rjk/
Lew Pitcher
2024-02-20 14:14:53 UTC
Permalink
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.

Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)

HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.

You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.

It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.

So, dont even try to "add an O_TEXT flag" to Linux. Just dont.
--
Lew Pitcher
"In Skills We Trust"
Paul Edwards
2024-02-20 14:25:05 UTC
Permalink
Post by Lew Pitcher
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.
Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)
HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.
You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.
The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.
Post by Lew Pitcher
It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.
Sorry - what is being invalidated? Can you give an
example?

BFN. Paul.
Lew Pitcher
2024-02-20 17:55:07 UTC
Permalink
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.
Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)
HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.
You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.
The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.
Lets take an accounting...

1) POSIX open() does not make any distinction between "text" and "binary"
files

2) Standard C / POSIX fopen() /does/ make a distinction between "text"
and "binary" files. POSIX issues a caveat regarding the mode
parameter that "The character 'b' shall have no effect, but is allowed
for ISO C standard conformance.", referring to the mode character "b"
specified by the ISO C standard as indicating that the file should be
opened for "binary" I/O, rather than the default "text" I/O.

3) You specifically reference the open() call, not the fopen() call.

So, you ask that the open() call, a call that, under Linux and other
POSIX-compatible operating systems, makes no distinction between binary
and text files, now recognize a flag indicating that the open() should
act apon a text file.

That leaves the Linux kernel developers with a choice to make: either
a) accept your request and add an O_TEXT flag that conditions the I/O
for text data, or
b) accept your request and add an O_TEXT flag that does nothing, or
c) ignore or deny your request

Choice (a) would drastically change the behaviour of the open()
call, and any existing code that uses open() to access a text file would
have to change to specify the O_TEXT flag. I believe that this would
be a no-starter, even if the default for O_TEXT would leave behaviour
as it currently is.

Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
It would be a header change that reserves a flag that would be ignored
by everyone. I believe that the maintainers would simply veto this
as being a change that makes no change.

So, choice (c) is my bet. The maintainers simply ignore (if they want
to be polite) or deny your proposal.
Post by Paul Edwards
Post by Lew Pitcher
It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.
Sorry - what is being invalidated? Can you give an
example?
Today, I can
open("/etc/passwd",O_RDONLY)
and read text from the text file /etc/passwd. I can
also
open("/sbin/init",O_RDONLY)
and read binary data from the binary file /sbin/init

If I understand correctly, you propose that code now
must
open("/etc/passwd",O_RDONLY | O_TEXT)
to read text from the text file /etc/passwd.

What would happen if, under your proposal, code still
executed
open("/etc/passwd",O_RDONLY)
? Could the code still read() text data, or would the
open() fail on something like EACCESS or EINVAL or
EOPNOTSUPP? What /would/ the consequences be of setting
O_TEXT?

Of course, if you answer that, under a POSIX-like system,
there would be /no/ consequences to O_TEXT, then you
have presented choice (b), a change that changes nothing.
--
Lew Pitcher
"In Skills We Trust"
Paul Edwards
2024-02-20 21:37:45 UTC
Permalink
Post by Lew Pitcher
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.
Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)
HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.
You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.
The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.
Lets take an accounting...
1) POSIX open() does not make any distinction between "text" and "binary"
files
2) Standard C / POSIX fopen() /does/ make a distinction between "text"
and "binary" files. POSIX issues a caveat regarding the mode
parameter that "The character 'b' shall have no effect, but is allowed
for ISO C standard conformance.", referring to the mode character "b"
specified by the ISO C standard as indicating that the file should be
opened for "binary" I/O, rather than the default "text" I/O.
3) You specifically reference the open() call, not the fopen() call.
So, you ask that the open() call, a call that, under Linux and other
POSIX-compatible operating systems, makes no distinction between binary
and text files, now recognize a flag indicating that the open() should
act apon a text file.
That leaves the Linux kernel developers with a choice to make: either
a) accept your request and add an O_TEXT flag that conditions the I/O
for text data, or
b) accept your request and add an O_TEXT flag that does nothing, or
c) ignore or deny your request
Choice (a) would drastically change the behaviour of the open()
call, and any existing code that uses open() to access a text file would
have to change to specify the O_TEXT flag. I believe that this would
be a no-starter, even if the default for O_TEXT would leave behaviour
as it currently is.
Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
It would be a header change that reserves a flag that would be ignored
by everyone.
Almost everyone.

PDOS/386 would use it.
Post by Lew Pitcher
I believe that the maintainers would simply veto this
as being a change that makes no change.
So, choice (c) is my bet. The maintainers simply ignore (if they want
to be polite) or deny your proposal.
Sure. Hence my question.

I'll take a random flag myself. Any suggestions?
Post by Lew Pitcher
Post by Paul Edwards
Post by Lew Pitcher
It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.
Sorry - what is being invalidated? Can you give an
example?
Today, I can
open("/etc/passwd",O_RDONLY)
and read text from the text file /etc/passwd. I can
also
open("/sbin/init",O_RDONLY)
and read binary data from the binary file /sbin/init
If I understand correctly, you propose that code now
must
open("/etc/passwd",O_RDONLY | O_TEXT)
to read text from the text file /etc/passwd.
What would happen if, under your proposal, code still
executed
open("/etc/passwd",O_RDONLY)
? Could the code still read() text data, or would the
open() fail on something like EACCESS or EINVAL or
EOPNOTSUPP? What /would/ the consequences be of setting
O_TEXT?
Of course, if you answer that, under a POSIX-like system,
there would be /no/ consequences to O_TEXT, then you
have presented choice (b), a change that changes nothing.
Correct. Nothing is invalidated.

It is only when run on a Windows-like system that
something happens.

BFN. Paul.
Lew Pitcher
2024-02-21 02:51:27 UTC
Permalink
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.
Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)
HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.
You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.
The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.
Lets take an accounting...
1) POSIX open() does not make any distinction between "text" and "binary"
files
2) Standard C / POSIX fopen() /does/ make a distinction between "text"
and "binary" files. POSIX issues a caveat regarding the mode
parameter that "The character 'b' shall have no effect, but is allowed
for ISO C standard conformance.", referring to the mode character "b"
specified by the ISO C standard as indicating that the file should be
opened for "binary" I/O, rather than the default "text" I/O.
3) You specifically reference the open() call, not the fopen() call.
So, you ask that the open() call, a call that, under Linux and other
POSIX-compatible operating systems, makes no distinction between binary
and text files, now recognize a flag indicating that the open() should
act apon a text file.
That leaves the Linux kernel developers with a choice to make: either
a) accept your request and add an O_TEXT flag that conditions the I/O
for text data, or
b) accept your request and add an O_TEXT flag that does nothing, or
c) ignore or deny your request
Choice (a) would drastically change the behaviour of the open()
call, and any existing code that uses open() to access a text file would
have to change to specify the O_TEXT flag. I believe that this would
be a no-starter, even if the default for O_TEXT would leave behaviour
as it currently is.
Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
It would be a header change that reserves a flag that would be ignored
by everyone.
Almost everyone.
PDOS/386 would use it.
If PDOS/386 needs it, then PDOS/386 probably should use it.

However, the needs of PDOS/386 aren't the needs of Linux or POSIX, and
neither Linux in particular, nor POSIX in general, need bother with
it.

If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard, then I suggest that you select the value and
treatment of your O_TEXT flag to work in absence, as POSIX code won't
have it or use it.

The alternative is to specify that such a flag /is/ required for PDOS/386,
which restricts you to using code written specifically for PDOS/386, and
not the more general-purpose and widely available POSIX standard.

[snip]


Best of luck.
--
Lew Pitcher
"In Skills We Trust"
Richard Kettlewell
2024-02-21 09:48:50 UTC
Permalink
Post by Lew Pitcher
If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard,
Not just source - that is easy to deal with.
The executable.
Executables won’t have your extra flag unless you’ve personally modified
and recompiled them.
Post by Lew Pitcher
The alternative is to specify that such a flag /is/ required for PDOS/386,
That's the alternative I am after.
Post by Lew Pitcher
which restricts you to using code written specifically for PDOS/386, and
It is only code that uses the new flag that will work nicely on
PDOS/386, yes.
How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?
--
https://www.greenend.org.uk/rjk/
Paul Edwards
2024-02-21 11:41:35 UTC
Permalink
Post by Richard Kettlewell
Post by Lew Pitcher
If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard,
Not just source - that is easy to deal with.
The executable.
Executables won’t have your extra flag unless you’ve personally modified
and recompiled them.
Yes, that's exactly what I will do. I will start
producing executables that have that flag.

At least MY executables will live up to MY
requirements.

Which is all I'm really after.

If someone else finds that useful, for the same
reasons I find that useful, great.

If I am alone in the world, that's fine too.

The latter state could change at a later date
though. You never know what will happen a
millenia from now.
Post by Richard Kettlewell
Post by Lew Pitcher
which restricts you to using code written specifically for PDOS/386, and
It is only code that uses the new flag that will work nicely on
PDOS/386, yes.
How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?
I only intend to run C90-compliant programs
(in this case, with a statically-linked C
runtime library).

fstat is not used by PDPCLIB, nor part of the
C90 standard.

lseek is used by PDPCLIB - that's how fseek()
is implemented.

According to C90, fseek() is not expected to
behave sensibly on text files.

And indeed - it won't (on PDOS/386 anyway; when
run on Linux it will be fine).

BFN. Paul.
Richard Kettlewell
2024-02-21 12:25:55 UTC
Permalink
Post by Richard Kettlewell
How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?
I only intend to run C90-compliant programs (in this case, with a
statically-linked C runtime library).
fstat is not used by PDPCLIB, nor part of the C90 standard.
Standard C doesn’t have open(), read() or write() either, but you seem
to want to modify the behaviour of those.
--
https://www.greenend.org.uk/rjk/
Paul Edwards
2024-02-21 12:43:32 UTC
Permalink
Post by Richard Kettlewell
Post by Richard Kettlewell
How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?
I only intend to run C90-compliant programs (in this case, with a
statically-linked C runtime library).
fstat is not used by PDPCLIB, nor part of the C90 standard.
Standard C doesn’t have open(), read() or write() either,
Indeed - because PDPCLIB's implementation of fopen()
needs to do the open syscall (INT 80H EAX = 5H).

fopen already knows whether the file is being
opened as text or binary, and I don't want that
information lost when I do that interrupt above.
Even though Linux (currently, and very likely
forever) won't use it, PDOS/386 has a use for it
(in the short term).
Post by Richard Kettlewell
but you seem to want to modify the behaviour of those.
I don't think that is an accurate statement.

What behavior, where?

Adding an extra flag doesn't change the behavior
of anything anywhere, immediately.

PDOS/386 would then be in a position to have a
particular behavior, but it's not really a
change, is it? It's new development/enhancement.

BFN. Paul.
Paul Edwards
2024-02-21 22:29:05 UTC
Permalink
Post by Paul Edwards
Post by Richard Kettlewell
Standard C doesn’t have open(), read() or write() either,
Indeed - because PDPCLIB's implementation of fopen()
needs to do the open syscall (INT 80H EAX = 5H).
fopen already knows whether the file is being opened as text or
binary, and I don't want that information lost when I do that
interrupt above. Even though Linux (currently, and very likely
forever) won't use it, PDOS/386 has a use for it (in the short term).
Post by Richard Kettlewell
but you seem to want to modify the behaviour of those.
I don't think that is an accurate statement.
What behavior, where?
You’ve been talking about doing translation _somewhere_,
Yes - when run under PDOS/386.

Currently the only Linux ELF program that runs
under PDOS/386 is a "hello world".

And I don't think anyone cares if I change the
behavior of that.

And that program doesn't even use open(), so is
not subject to my proposed change.

I guess I am talking about changing FUTURE
behavior of a FUTURE program.

But I thought you were implying I was going
to break something.
and cited
Cygwin as an example in your first posting, and have been consistently
using the flag name from Cygwin. Cygwin’s O_TEXT causes read() and
write() to translate between newline conventions, so if you meant
something else then it was a confusing decision to use Cygwin as an
example.
Cygwin is a correct example, I believe.

And Cygwin doesn't change the behavior of Linux ELF
executables either.

Nor does it change the behavior of existing POSIX
source code.
If, in fact, you don’t want to add translation to read() and write()
I do.

I guess it's just a semantic debate as to whether
this constitutes a change in behavior.

All existing Linux ELF programs are unchanged.

Even new Linux ELF programs won't change when
run under Linux.

The flag only takes effect when the ELF is run
on a competing system.

BFN. Paul.
Richard Kettlewell
2024-02-22 09:57:29 UTC
Permalink
Post by Paul Edwards
Cygwin is a correct example, I believe.
And Cygwin doesn't change the behavior of Linux ELF
executables either.
AFAIK it can’t run them at all, so I don’t know why that would be
relevant.
Post by Paul Edwards
Nor does it change the behavior of existing POSIX source code.
Your use case was C90 programs, not POSIX programs, last time you
mentioned it. Has that changed now?
Post by Paul Edwards
If, in fact, you don’t want to add translation to read() and write()
I do.
OK, so an ELF executable (that had been modified to use your
hypothetical O_TEXT flag) would get newline translation from read() and
write() when run on your toy OS, but not when run on a real Linux
kernel. There’s the change in behavior.

At any rate we’re back to the issue that you say want to add
Cygwin-style translation to read() and write(), but you also say that
you will only be running programs written in C90, which doesn’t have
open(), so nothing will ever pass O_TEXT and the translation will never
be activated.
Post by Paul Edwards
I guess it's just a semantic debate as to whether this constitutes a
change in behavior.
This would go a lot quicker if you didn’t engage in “semantic debates”.
--
https://www.greenend.org.uk/rjk/
Paul Edwards
2024-02-22 11:23:26 UTC
Permalink
Post by Richard Kettlewell
Post by Paul Edwards
Cygwin is a correct example, I believe.
And Cygwin doesn't change the behavior of Linux ELF
executables either.
AFAIK it can’t run them at all, so I don’t know why that would be
relevant.
That's exactly the point - it doesn't run them.
No behavior is being changed by Cygwin either.
Post by Richard Kettlewell
Post by Paul Edwards
Nor does it change the behavior of existing POSIX source code.
Your use case was C90 programs, not POSIX programs, last time you
mentioned it. Has that changed now?
I am the vendor of a C90 runtime library for Linux (PDPCLIB).

fopen() necessarily does an open syscall.
Post by Richard Kettlewell
Post by Paul Edwards
If, in fact, you don’t want to add translation to read() and write()
I do.
OK, so an ELF executable (that had been modified to use your
hypothetical O_TEXT flag) would get newline translation from read() and
write() when run on your toy OS, but not when run on a real Linux
kernel. There’s the change in behavior.
Well you can call it that if you want. But as I said,
I think it is odd to call something a change in behavior
when the software that would be "changing" hasn't even
been written yet. Or at least, the INT 80H open() processing
of PDOS/386 hasn't been written yet. Only write to stdout
has been written so far. And exit.

Also, when you say "the ELF executable that has been
modified", I would use the word "built" rather than
"modified". I am building new ELF executables according
to a hypothetical version of POSIX/Linux that includes
a new O_TEXT definition.
Post by Richard Kettlewell
At any rate we’re back to the issue that you say want to add
Cygwin-style translation to read() and write(), but you also say that
you will only be running programs written in C90, which doesn’t have
open(), so nothing will ever pass O_TEXT and the translation will never
be activated.
PDPCLIB will do that.

I clearly haven't done a very good job of explaining
what I want.

I am organizing contact with the "Austin Group" (I've
become a member and downloaded the latest POSIX draft).

Is there any wording I should use to avoid the several
days of confusion we've had here?

When I wrote my original message I thought it was a very
simple suggestion, using the widely-known Cygwin prior art.

Instead, I'm not even sure my proposal is understood.
Post by Richard Kettlewell
Post by Paul Edwards
I guess it's just a semantic debate as to whether this constitutes a
change in behavior.
This would go a lot quicker if you didn’t engage in “semantic debates”.
I wasn't attempting to. I was just trying to identify
the point of contention as semantics, not a technical
issue.

It's still not clear to me that it really is semantics,
because you are using language that I wouldn't use.

BFN. Paul.
Paul Edwards
2024-02-23 02:15:56 UTC
Permalink
Post by Paul Edwards
Post by Richard Kettlewell
Post by Paul Edwards
Cygwin is a correct example, I believe.
And Cygwin doesn't change the behavior of Linux ELF
executables either.
AFAIK it can’t run them at all, so I don’t know why that would be
relevant.
That's exactly the point - it doesn't run them.
No behavior is being changed by Cygwin either.
The behavior of read() and write() is changed by the O_TEXT flag in
Cygwin.
Sure - that's existing Cygwin that has *already
changed*. I'm not proposing a change to anything
that already exists.
Post by Paul Edwards
Post by Richard Kettlewell
Post by Paul Edwards
Nor does it change the behavior of existing POSIX source code.
Your use case was C90 programs, not POSIX programs, last time you
mentioned it. Has that changed now?
I am the vendor of a C90 runtime library for Linux (PDPCLIB).
fopen() necessarily does an open syscall.
If a file was opened with fopen() then the stdio functions (putc, fread,
etc) can do the newline translation.
I don't WANT to do newline translation when *my*
binary *runs on Linux*.

I want my binaries to behave like any normal Linux program.

So I can't translate in the C library, unless the C
library does some sort of special call to detect the
environment and change behavior accordingly.

I think O_TEXT (with Cygwin prior art) is the neater
solution to this logical problem.
No need to mess with the behavior
read() and write().
No need to mess with the behavior of read() and
write() on WHICH SYSTEM?

PDOS/386 is the only thing that would be doing
any messing.

And that's far less mess than the alternative.
But I think we’ve been round that loop and for
unknown reasons you still seem to want to do it the hard way.
I don't consider this to be the hard way.

Can you suggest a syscall that I could use to detect
whether I am running under true Linux or PDOS/386
so that I can get the C library to adjust appropriately?

Also, the C library (as with Cygwin) may need to adjust
on a file-by-file basis. ie a syscall needs to be done
"do you want translation to be done on this particular
filename?". open() doesn't have the ability to return
such a status, so I would need to do a similar syscall
to open to find the translation requirements.

And this overhead needs to be added to (my) Linux programs
even when running on Linux - overhead for no purpose,
when a simple flag (that is ignored) is all that is required.
Post by Paul Edwards
Post by Richard Kettlewell
Post by Paul Edwards
If, in fact, you don’t want to add translation to read() and write()
I do.
OK, so an ELF executable (that had been modified to use your
hypothetical O_TEXT flag) would get newline translation from read() and
write() when run on your toy OS, but not when run on a real Linux
kernel. There’s the change in behavior.
Well you can call it that if you want. But as I said, I think it is
odd to call something a change in behavior when the software that
would be "changing" hasn't even been written yet. Or at least, the
INT 80H open() processing of PDOS/386 hasn't been written yet. Only
write to stdout has been written so far. And exit.
Once again, this would go a lot quicker if you didn’t engage in
“semantic debates”.
I'm not deliberately engaging in semantic debates.

I'm just saying that if that's your definition, then
fine, yes, you're right, by that definition.
Post by Paul Edwards
Also, when you say "the ELF executable that has been modified", I
would use the word "built" rather than "modified". I am building new
ELF executables according to a hypothetical version of POSIX/Linux
that includes a new O_TEXT definition.
If it’s unmodified then it won’t set O_TEXT.
If *WHAT* is unmodified?

An existing ELF binary or one that I will build in
the future?
Post by Paul Edwards
Post by Richard Kettlewell
At any rate we’re back to the issue that you say want to add
Cygwin-style translation to read() and write(), but you also say that
you will only be running programs written in C90, which doesn’t have
open(), so nothing will ever pass O_TEXT and the translation will
never be activated.
PDPCLIB will do that.
If you control the C library then you can do the translation in the
<stdio.h> implementation; there is no need to mess with the behavior of
read() and write().
Not true. The translation needs to be conditional.
So I need a different set of infrastructure to do
translation in the C library.
Post by Paul Edwards
I clearly haven't done a very good job of explaining what I want.
Indeed.
And I'm not doing that deliberately either.

I'm a native English speaker, but this is not the
first time I have failed to convey my message.

Sorry about that, but I don't know where I am
going wrong.

Maybe I can express it in C code/pseudocode.


Solution 1:

fopen()
{
if (strcmp(mode, "r")
{
flag |= O_TEXT; /* cannot be zero!!! */
}
syscall(syscall_open, name, flag);
}


Solution 2:

fopen()
{
if (strcmp(mode, "r")
{
rc = syscall(syscall_os_check);
if (rc == PDOS386) /* as opposed to GENUINE LINUX */
{
rc = syscall(syscall_check_translation, filename);
if (rc == TRANSLATION_REQUIRED)
{
FILE.translation_flag = 1;
}
}
}
}

fwrite()
{
if (FILE.translation_flag)
{
for (...) /* add CRs */
}
}


Let me know if the pseudocode is understandable,
or needs to be fleshed out in even more detail.

BFN. Paul.
Richard Kettlewell
2024-02-23 10:25:30 UTC
Permalink
No need to mess with the behavior read() and write().
No need to mess with the behavior of read() and write() on WHICH
SYSTEM?
Any system.
But I think we’ve been round that loop and for unknown reasons you
still seem to want to do it the hard way.
I don't consider this to be the hard way.
Can you suggest a syscall that I could use to detect
whether I am running under true Linux or PDOS/386
so that I can get the C library to adjust appropriately?
uname()
Specifically ones linked with PDPCLIB where I have
done very specific things (which I control) so that
THE BINARIES work on BOTH Linux and PDOS/386.
Do you get that this is a weird requirement? You could just compile your
code twice, one for Linux and once for your toy platform.
--
https://www.greenend.org.uk/rjk/
Paul Edwards
2024-02-23 11:31:42 UTC
Permalink
Post by Richard Kettlewell
No need to mess with the behavior read() and write().
No need to mess with the behavior of read() and write() on WHICH
SYSTEM?
Any system.
There is a need, or at least, desire, to mess
with the behavior of read and write on PDOS/386,
for the exact same reason that Cygwin does.

It has no impact on anyone else.
Post by Richard Kettlewell
But I think we’ve been round that loop and for unknown reasons you
still seem to want to do it the hard way.
I don't consider this to be the hard way.
Can you suggest a syscall that I could use to detect
whether I am running under true Linux or PDOS/386
so that I can get the C library to adjust appropriately?
uname()
Thanks. I've just tested that (syscall 122).

***@kerravon2-pc:/home/kerravon/w2kshare# ./pdptest.exe
welcome to pdptest
buf is XLinuxX
buf + 65 is Xkerravon2-pcX
buf + 65 * 2 is X6.5.0-18-genericX
buf + 65 * 3 is X#18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7
11:40:03 UTC 2X
buf + 65 * 4 is Xx86_64X
buf + 65 * 5 is X(none)X
m1 is 00000000

It looks to me like I need a buffer of size 6 * 65,
but I can add another 200 bytes and hope that is
enough.

It's not ideal, but it'll probably be sufficient.

And perhaps I could fudge this call so that on
PDOS/386 it returns 0x1234 as a magic number
instead of 0, so that I know it is PDOS/386.

That way I don't need to do a strcmp to a
particular system name. As I guess ultimately
I am trying to develop an alternative to POSIX.
So this could indicate that the alternative is
in effect.

And then if I am on a PDOS/386 (or POSIX-alternate)
system when I am opening a text file I can include a
flag of 0x8000 0000 to the open syscall.
Post by Richard Kettlewell
Specifically ones linked with PDPCLIB where I have
done very specific things (which I control) so that
THE BINARIES work on BOTH Linux and PDOS/386.
Do you get that this is a weird requirement? You could just compile your
code twice, one for Linux and once for your toy platform.
Sure - in fact I could just use the Win32 binaries.

But if I have to do that, I can no longer advertise
it as a "mini-Linux clone" that runs "properly-built"
Linux executables.

Where "properly-built" is something I just make up now
and roughly means "supports both POSIX and POSIX-alternate
systems".

BFN. Paul.
Paul Edwards
2024-02-28 13:18:49 UTC
Permalink
Note that I have a new pdos.zip at http://pdos.org
that implements this and other things (like a dummy
fork and trimmed exec so that system() from the
Linux app works on PDOS/386).

I went with O_TEXT of 0x4000 0000 so that 0x8000 0000
could be reserved to mean "more flags available in
another variable" or whatever.

Elsewhere someone told me that even if POSIX defined
O_TEXT, in Linux they could just set that to 0, so
it was Linux I needed, not POSIX.

So I wrote to the linux-kernel mailing list asking
for suggestions, but there was no reply so I went
ahead with the above.

Proof of concept for all features (another one is
command line with no Linux stack) can be found here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/src/int80.c

And it runs this Linux test program:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/makefile.lnp

That concludes my immediate interest in Linux,
and I'm on to OS/2 that needs to be fleshed
out more.

BFN. Paul.
Jasen Betts
2024-02-29 13:02:56 UTC
Permalink
Post by Paul Edwards
Post by Richard Kettlewell
Post by Paul Edwards
Cygwin is a correct example, I believe.
And Cygwin doesn't change the behavior of Linux ELF
executables either.
AFAIK it can’t run them at all, so I don’t know why that would be
relevant.
That's exactly the point - it doesn't run them.
No behavior is being changed by Cygwin either.
Post by Richard Kettlewell
Post by Paul Edwards
Nor does it change the behavior of existing POSIX source code.
Your use case was C90 programs, not POSIX programs, last time you
mentioned it. Has that changed now?
I am the vendor of a C90 runtime library for Linux (PDPCLIB).
As the vendor you get to choose the constants used in the header file
provided for compiling programs to link against your library.

you loose compatability with other standard libraries. but maybe
that's not a problem
Post by Paul Edwards
I think it is odd to call something a change in behavior
when the software that would be "changing" hasn't even
been written yet. Or at least, the INT 80H open() processing
of PDOS/386 hasn't been written yet. Only write to stdout
has been written so far. And exit.
--
Jasen.
🇺🇦 Слава Україні
J.O. Aho
2024-02-22 12:48:23 UTC
Permalink
Post by Richard Kettlewell
Post by Paul Edwards
I guess it's just a semantic debate as to whether this constitutes a
change in behavior.
This would go a lot quicker if you didn’t engage in “semantic debates”.
For me it whole feels like a repeat of systemd when they stupidly picked
a namespace already used by the kernel and wanted the kernel to change
nem of their namespace.

The whole can be solved in the header to define missing macro.
--
//Aho
Paul Edwards
2024-02-22 13:49:19 UTC
Permalink
Post by J.O. Aho
Post by Richard Kettlewell
Post by Paul Edwards
I guess it's just a semantic debate as to whether this constitutes a
change in behavior.
This would go a lot quicker if you didn’t engage in “semantic debates”.
For me it whole feels like a repeat of systemd when they stupidly picked
a namespace already used by the kernel and wanted the kernel to change
nem of their namespace.
The whole can be solved in the header to define missing macro.
Define it to what? That's my original question.

And by who?

BFN. Paul.
Paul Edwards
2024-02-23 01:56:16 UTC
Permalink
Post by Paul Edwards
Post by J.O. Aho
Post by Richard Kettlewell
Post by Paul Edwards
I guess it's just a semantic debate as to whether this constitutes a
change in behavior.
This would go a lot quicker if you didn’t engage in “semantic debates”.
For me it whole feels like a repeat of systemd when they stupidly picked
a namespace already used by the kernel and wanted the kernel to change
nem of their namespace.
The whole can be solved in the header to define missing macro.
Define it to what? That's my original question.
That's up to whomever defines it.
Post by Paul Edwards
And by who?
That would be you. And /not/ by Linux.
That's not correct.

I can't properly define it myself. If I define it
myself, to an arbitrary value, then the ELF executable
will do strange things now or in the future when run
on Linux.

ie if I take over an existing flag or flags, or
choose a currently unused bit.

Again - I've clearly done a really really crappy
job of explaining what I want.

I don't know what I'm doing wrong.

Is there anyone who actually understands my
suggestion who can translate English to English?

I'll need a translated version for when I speak
to the Austin group otherwise I'll be going around
the exact same circle.

It's possible that I have some "unstated assumptions".
I have found that a problem in the past. But I don't
know what they are until after a lot of discussion.

I can take a guess though.

My goal for PDOS/386 is not to run *ANY* *EXISTING*
Linux ELF binaries.

My goal is to run *SPECIFIC* *FUTURE* Linux ELF
binaries.

Specifically ones linked with PDPCLIB where I have
done very specific things (which I control) so that
THE BINARIES work on BOTH Linux and PDOS/386.

I'm not attempting to create a Linux clone. I'm only
trying to create a very limited clone.

And I'm not trying to get CRs on Linux. I'm only
trying to get CRs on PDOS/386.

Again - I thought I explained it adequately, but of
course I always think that, but this is not the first
time I have been mistaken about what is "clear".

BFN. Paul.
J.O. Aho
2024-02-23 09:25:16 UTC
Permalink
Post by Paul Edwards
Post by Paul Edwards
And by who?
That would be you. And /not/ by Linux.
I can't properly define it myself. If I define it
myself, to an arbitrary value, then the ELF executable
will do strange things now or in the future when run
on Linux.
Try 0x0

You need to handle it in you own library pdpclib, do not expect others
to change for you think providing a file that isn't the one on disk is ok.
--
//Aho
Paul Edwards
2024-02-23 10:30:54 UTC
Permalink
Post by J.O. Aho
Post by Paul Edwards
Post by Paul Edwards
And by who?
That would be you. And /not/ by Linux.
I can't properly define it myself. If I define it
myself, to an arbitrary value, then the ELF executable
will do strange things now or in the future when run
on Linux.
Try 0x0
You need to handle it in you own library pdpclib, do not expect others
to change for you think providing a file that isn't the one on disk is ok.
If it boils down to that, then I will do one of
the things I already mentioned.

I will define it to 0x8000 0000 or 0x4000 0000
or alter the definition of an existing flag
like NOATIME.

It will achieve what I want, even though it is
a bit odd when running on real Linux.

Any suggestions within those parameters?

Otherwise I will just choose one of those things
at random and carry on my merry way.

Probably I will chose 0x8000 0000 as Linux seems
to be working their way up the bits, so I will
take what I need working my way down the bits.

What I don't know yet is whether Linux will throw
an error for the unknown flag (0x8000 0000) or
whether they will in the future put in such a
check.

If they do, I then need to switch to commandeering
NOATIME or whatever and rebuilding my software.

BFN. Paul.
Jasen Betts
2024-02-29 12:51:35 UTC
Permalink
Post by Richard Kettlewell
Post by Lew Pitcher
If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard,
Not just source - that is easy to deal with.
The executable.
Executables won’t have your extra flag unless you’ve personally modified
and recompiled them.
Some source targeted for DOS will have O_TEXT, Turbo C had this feature
in 1988 but if you go there you probably need to implement fmode also,
Post by Richard Kettlewell
Post by Lew Pitcher
The alternative is to specify that such a flag /is/ required for PDOS/386,
That's the alternative I am after.
Post by Lew Pitcher
which restricts you to using code written specifically for PDOS/386, and
It is only code that uses the new flag that will work nicely on
PDOS/386, yes.
How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?
They don't play nicely don't on DOS or Windows (or UTF-8 etc...)

This is probably not needed. I'm faily sure standards allow ftell and
fseek to deal in cookies on text files.
--
Jasen.
🇺🇦 Слава Україні
Paul Edwards
2024-02-21 13:54:02 UTC
Permalink
Post by Lew Pitcher
b) accept your request and add an O_TEXT flag that does nothing, or
Choice (b) is ... useless. It recognizes a non-issue, and does
nothing. It would be a header change that reserves a flag that would
be ignored by everyone. I believe that the maintainers would simply
veto this as being a change that makes no change.
This would be useless, true. But it would be harmless. It could be
defined simply as
#define O_TEXT 0
No. It can't be that. It needs to be non-zero so
that I can detect it in a non-Linux environment.
I think what the OP is proposing is option (b) above: allow an O_TEXT
identifier having no effect.
No effect ... on Linux.

An effect ... on PDOS/386.
To the OP: a solution to your problem is simple. Put the following
#ifndef O_TEXT
#define O_TEXT 0
#endif
No, that doesn't solve my problem.

When the ELF executable is run on PDOS/386, I will
have no way of knowing that this open is text, and
thus PDOS/386 (the OS) should add CRs whenever it
sees a LF.

Because we're beyond the point when anyone else
would add the CR.

BFN. Paul.
Lew Pitcher
2024-02-21 20:28:35 UTC
Permalink
On Wed, 21 Feb 2024 21:54:02 +0800, Paul Edwards wrote:
[snip]
Post by Paul Edwards
When the ELF executable is run on PDOS/386, I will
have no way of knowing that this open is text, and
thus PDOS/386 (the OS) should add CRs whenever it
sees a LF.
ISTM that you are looking at this from the wrong end.

POSIX open() does not make a distinction between a "text"
and a "binary" file; all files are "binary". Programs
(including Linux programs) that are coded to the POSIX
standard already accommodate this. So, any code you port
from a Linux environment (either source code or ELF
executable) is already in a state to read and write
a binary file. Given this, then there is no need for
an O_TEXT flag for a Linux program, as no Linux program
expects open() to provide special conditions for text
data.

Given this, there only one problem to resolve: how to
allow for text data to be interchanged transparently
between the MSDOS/Windows side of your PDOS/386 and
the Linux side.

And, that's not the problem you are chasing ATM.

[snip]
--
Lew Pitcher
"In Skills We Trust"
Paul Edwards
2024-02-21 22:22:13 UTC
Permalink
Post by Lew Pitcher
[snip]
Post by Paul Edwards
When the ELF executable is run on PDOS/386, I will
have no way of knowing that this open is text, and
thus PDOS/386 (the OS) should add CRs whenever it
sees a LF.
ISTM that you are looking at this from the wrong end.
POSIX open() does not make a distinction between a "text"
and a "binary" file; all files are "binary". Programs
Sure. And I'm talking about changing POSIX to
add the distinction, for anyone who has a use
for such a thing.
Post by Lew Pitcher
(including Linux programs) that are coded to the POSIX
standard already accommodate this. So, any code you port
from a Linux environment (either source code or ELF
executable) is already in a state to read and write
a binary file.
This would be new code obviously, and include
an O_TEXT.
Post by Lew Pitcher
Given this, then there is no need for
an O_TEXT flag for a Linux program, as no Linux program
expects open() to provide special conditions for text
data.
I want to be able to WRITE such a Linux program.
Post by Lew Pitcher
Given this, there only one problem to resolve: how to
allow for text data to be interchanged transparently
between the MSDOS/Windows side of your PDOS/386 and
the Linux side.
PDOS/386 *will be Linux*. A mini-clone. Also a
mini-clone of OS/2. It's already a mini-clone
of Windows, and sort of a clone of MSDOS too.

All on a 360k floppy.

The only question is - on this all-encompassing OS -
are text files going to follow DOS (CP/M) convention
or Unix convention?

And currently I am going for CP/M, which means my
Linux executables need to provide the appropriate
information so that they run seamlessly in this
CP/M-standard environment.

One day I'll probably do the reverse. And since it
is the C library that is adding CRs for Windows,
it will require a similar enhancement for the
Windows CreateFile() interface to pass a "text"
flag so that PDOS/386 can STRIP the CRs that will
be flowing through.

But that looks even more difficult:

https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea

BFN. Paul.
Richard Kettlewell
2024-02-20 15:45:36 UTC
Permalink
Post by Lew Pitcher
Post by Paul Edwards
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I'd like to add an O_TEXT flag to Linux.[snip]
Any suggestions?
Yes, one. Dont.
Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)
AFAIK it’s not that low level. I can see no sign of it in CreateFile,
ReadFile, etc, which are approximately equivalent to open(), read() etc
in the Windows API. I think the implementation is in the C runtime
(effectively the Libc).

Cygwin does not appear rely on Windows for the implementation of its
O_TEXT and O_BINARY - you can see the translation code in
fhandler_base::read.
--
https://www.greenend.org.uk/rjk/
Lew Pitcher
2024-02-20 19:34:14 UTC
Permalink
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I am
still missing the same thing that Cygwin was
missing, and added O_TEXT to solve.
And, what would that be? What does the Cygwin
O_TEXT flag on open(2) do?

[snip]
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?

If you just want to keep source compatibility, I don't see
any problem with using a system-specific predefined macro
to implement some conditional compilation. Something like

#ifndef O_TEXT
#define O_TEXT 0
#endif

open("/etc/passwd",O_RDONLY | O_TEXT);

HTH
--
Lew Pitcher
"In Skills We Trust"
Paul Edwards
2024-02-20 21:40:33 UTC
Permalink
Post by Lew Pitcher
Post by Paul Edwards
Hi.
Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.
[snip]
Post by Paul Edwards
I am
still missing the same thing that Cygwin was
missing, and added O_TEXT to solve.
And, what would that be? What does the Cygwin
O_TEXT flag on open(2) do?
It will add CRs to such files on a write().
Post by Lew Pitcher
[snip]
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?
The latter. Functionality on PDOS/386, not Linux.

And not Linux source. Linux binaries.
Post by Lew Pitcher
If you just want to keep source compatibility, I don't see
any problem with using a system-specific predefined macro
to implement some conditional compilation. Something like
#ifndef O_TEXT
#define O_TEXT 0
#endif
open("/etc/passwd",O_RDONLY | O_TEXT);
Not what I'm after.

BFN. Paul.
Paul Edwards
2024-02-21 11:57:13 UTC
Permalink
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?
The latter. Functionality on PDOS/386, not Linux.
Think you need to look into the windows source code, as open() already
exists in windows
There is no such syscall. kernel32 exports CreateFile.

Regardless - Windows won't run the Linux ELF binary -
not even the one I produce myself - regardless. Although
if you install WSL it would, but that's installing
Linux - and of course Linux programs run under Linux.

It is PDOS/386 that will run it. And receive the
INT 80H. And check for the flag.

Because PDOS/386 is sort of a DOS/Windows clone,
all files are expected to use CRLF as line terminators.
That's the user requirements, basically.

So I'm trying to get my Linux ELF binaries (ones that
I build myself) to fit into that existing user
requirement.
and those cygwin don't need to reimplement that, it
only provides things that do not exists in windows but is part of POSIX.
It is Cygwin that invented O_TEXT, not Windows.
To satisfy that same user requirement above.
Post by Paul Edwards
And not Linux source. Linux binaries.
- a way to run native Linux apps on Windows. You must rebuild your
application from source if you want it to run on Windows.
I know.

That's why I'm not using Cygwin, I'm using PDOS/386.

Which ALREADY runs a hello world Linux ELF binary.

(unlike Cygwin - which never did)
If you want to make changes to POSIX API, you need to start contacting
Austin Group, with time changes will appear in Linux.
https://en.wikipedia.org/wiki/Austin_Group

Ok, thanks.
I doubt you would
have more luck with the Austin Group than convincing Linus to add a
patch to add something that Linux don't need.
Ok. I'll try the Austin Group first though. As you
noted, this isn't for Linux. It's for (quibbling
aside) a competitor of Linux, so the Austin Group
would be more appropriate.

BFN. Paul.
J.O. Aho
2024-02-21 12:52:40 UTC
Permalink
Post by Paul Edwards
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?
The latter. Functionality on PDOS/386, not Linux.
Think you need to look into the windows source code, as open() already
exists in windows
There is no such syscall. kernel32 exports CreateFile.
_sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.
Post by Paul Edwards
Because PDOS/386 is sort of a DOS/Windows clone,
all files are expected to use CRLF as line terminators.
That's the user requirements, basically.
Why don't you ask Microsoft to make ms-windows POSIX compatible, then
this issue would disappear at once ;)
Post by Paul Edwards
So I'm trying to get my Linux ELF binaries (ones that
I build myself) to fit into that existing user
requirement.
That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
if they are missing when they want to support multiple OS:es. They don't
expect that all OS has AmigaOS dos library implemented to open files
IDOS->Open("c:dir", MODE_OLDFILE)
--
//Aho
Paul Edwards
2024-02-21 14:52:24 UTC
Permalink
Post by J.O. Aho
Post by Paul Edwards
Think you need to look into the windows source code, as open() already
exists in windows
There is no such syscall. kernel32 exports CreateFile.
_sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.
Thanks for the pointer.

That allowed me to find this:

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen?view=msvc-170

https://learn.microsoft.com/en-us/cpp/c-runtime-library/text-and-binary-mode-file-i-o?view=msvc-170

And those functions exist here:

C:\WINNT\system32>odwin -x msvcrt.dll | grep -i open
[ 218] _fdopen
[ 242] _fsopen
[ 403] _open
[ 404] _open_osfhandle
[ 414] _popen
[ 445] _sopen
[ 527] _wfdopen
[ 534] _wfopen
[ 535] _wfreopen
[ 536] _wfsopen
[ 547] _wopen
[ 550] _wpopen
[ 558] _wsopen
[ 620] fopen
[ 628] freopen

C:\WINNT\system32>

(on my Windows 2000 system)

However, do note that this msvcrt.dll is undocumented
and nominally not meant to be used directly, and only
as part of Visual Studio (and even then, it's no longer
called that - it gets a version number added).

So this is very different from a kernel32.dll equivalent
of a syscall.

However, it does at least give prior art - on top of
Cygwin.

So it is something that I can potentially go to the
Austin Group with.
Post by J.O. Aho
Post by Paul Edwards
Because PDOS/386 is sort of a DOS/Windows clone,
all files are expected to use CRLF as line terminators.
That's the user requirements, basically.
Why don't you ask Microsoft to make ms-windows POSIX compatible, then
this issue would disappear at once ;)
Well that's what everyone has been saying for decades
now, and Microsoft wins every time.

So you can continue fighting that war - no problem.

However, I do want to approach it from the other
direction - a Windows clone.

And I've been working on mine for decades too.
Post by J.O. Aho
Post by Paul Edwards
So I'm trying to get my Linux ELF binaries (ones that
I build myself) to fit into that existing user
requirement.
That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
if they are missing when they want to support multiple OS:es.
That solves a different problem a different way.
Post by J.O. Aho
They don't
expect that all OS has AmigaOS dos library implemented to open files
IDOS->Open("c:dir", MODE_OLDFILE)
Well that's an interesting analogy.

One that I have already visited.

So, let's start with PCs with the same processor,
in this case 68000. Atari is a competitor to the
Amiga. Can the Atari run Amiga software?

The equivalent of "everyone should be POSIX" is
"everyone should be Atari". And that's what Atari
thinks. And the Amiga will win every time.

So, my approach - can the Atari run Amiga software?
At least C90-compliant software.

And the answer is - no. At their heart, the Amiga
executables work (without DLLs) by hardcoding the
number 4. All the Amigados functions hang off that.

And on the Atari, address 4 cannot be used.

So should we give up?

No.

First of all, we change the Amiga applications - in
a way that doesn't affect them running on the Amiga.

See here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/amistart.asm

I carry an alternate sysbase in d7. It won't be set
when running under genuine AmigaOS, but when run on
a competitor - perhaps not Atari's OS, but some
other OS running on Atari hardware - like, PDOS/Atari
or something like that - then d7 is set to a register
that points to memory that the Atari can actually
address properly.

And then those - select - AmigaOS executables - basically
ones compiled against PDPCLIB which is the only C library
I know of that implements the d7 protocol - which I only
created a couple of years ago - will run on alternate
hardware/OS.

And now I'm doing the same with Linux executables.

First I change the Linux executables with a view to
getting them to run on competing vendor platforms,
build those executables, and then later bring the
competing platform up to speed.

The d7 Amiga "standard" has been invented years (so
far) in advance of a competitor actually arising.

But that's for a different reason - I'm not that
interested in the Amiga API.

But I am interested in the OS/2 2.0 API.

And indeed - I have already prepared - or in the
midst of preparing - modifications there too.
Instead of depending on the 16-bit thunk libraries
to exist, I'm getting raw keyboard input by using
DosDevIOCtl. You can see that here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/stdio.c#l2184

So now I am getting into position for PDOS/386 to
support the 32-bit MSDOS API (which I sort of
created myself because Microsoft didn't do that),
Win32, OS/2 2.0 and Linux ELF.

All in the same OS, all with very little code to
support the different OSes, and the OS hopefully
fits on a 360k floppy even with that extra
functionality (or should come very close, anyway -
noting that it isn't a design goal to fit on a
360k floppy - 720k would be period accurate I think).

BFN. Paul.
Jasen Betts
2024-02-29 13:16:46 UTC
Permalink
Post by Paul Edwards
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?
The latter. Functionality on PDOS/386, not Linux.
Think you need to look into the windows source code, as open() already
exists in windows
There is no such syscall. kernel32 exports CreateFile.
Regardless - Windows won't run the Linux ELF binary -
not even the one I produce myself - regardless. Although
if you install WSL it would, but that's installing
Linux - and of course Linux programs run under Linux.
It is PDOS/386 that will run it. And receive the
INT 80H. And check for the flag.
so make you library figure out which OS is running and
remap the flags given to the to open() function call as
apropriate when translating to int80
--
Jasen.
🇺🇦 Слава Україні
Paul Edwards
2024-03-02 15:31:49 UTC
Permalink
Post by Jasen Betts
Post by Paul Edwards
Post by Paul Edwards
Post by Lew Pitcher
Post by Paul Edwards
And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.
Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?
The latter. Functionality on PDOS/386, not Linux.
Think you need to look into the windows source code, as open() already
exists in windows
There is no such syscall. kernel32 exports CreateFile.
Regardless - Windows won't run the Linux ELF binary -
not even the one I produce myself - regardless. Although
if you install WSL it would, but that's installing
Linux - and of course Linux programs run under Linux.
It is PDOS/386 that will run it. And receive the
INT 80H. And check for the flag.
so make you library figure out which OS is running and
remap the flags given to the to open() function call as
apropriate when translating to int80
Why would anything need to be remapped? Whenever
Linux binaries are run, I use standard Linux
values for everything in every system call. With
one single exception, which is an addition, not
a change - I add the O_TEXT of 0x4000 0000, which
I want ignored on Linux (only), and is indeed
ignored (on Linux).

Anyway, PDOS/386 now runs 32-bit MSDOS (by some
definition), Win32, Linux and OS/2 binaries. A
subset of valid binaries. Enough to run my
toolchain, and use my toolchain to rebuild my
toolchain.

OS/2 still has some gaps in the toolchain though.
I don't have a linker to build OS/2 executables
and I don't have the microemacs editor working
for OS/2 32-bit yet. Both are being worked on
currently.

And none of the executables I produce are dependent
on the registers or stack at entry, which clears
the way to introduce the PDOS-generic ABI, where
instead of DLLs, you are given a pointer to a
structure with callback functions for a C library.

BTW, another development was UC386L, a 50k Linux
executable, with no external dependencies, not
even libc, is capable of running that same subset
of valid Win32 and OS/2 executables.

BFN. Paul.

Loading...