[Lazarus] SdpoSerial Tx and Rx buffers?

classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Bo Berglund
On Wed, 16 Feb 2011 10:50:14 +0100, Michael Schnell
<[hidden email]> wrote:

>On 02/14/2011 08:32 PM, Bo Berglund wrote:
>>
>> In a blocking design this wait loop would of course not have been
>> needed becaus ethe Write would not return until all data have been
>> written to the output.
>But here the program's GUI stops working, as the software hangs in the
>blocking read or write,
>
>So IMHO, the only decent way to handle a serial interface is doing
>blocking communications calls in threads and fire events in the main
>thread (e.g. by Application.QueuAsyncCall (avoiding the "Windowish"
>PostMessage() stuff ) ) .
>
>And here, of course, you do need buffers to transfer data between the
>main thread and the threads

Exactly...


--
Bo Berglund
Developer in Sweden


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Hans-Peter Diettrich
In reply to this post by José Mejuto
José Mejuto schrieb:

> MS> Regarding sending, there still is a FiFo (usually some 4..64  Bytes ) in
> MS> the serial hardware (hundreds or thousands of bytes) and a software Fifo
> MS> in the OS driver. So the software can't tell when something has left the
> MS> serial port. For such deeply embedded issues  the specifics of the
> MS> driver and hardware needs to be considered
>
> AFAIK the FIFO is in the receive, in sent the FIFOs are filled but
> function does not return until the hardware sends the last byte.

I'm not sure, but IMO a FIFO for outgoing data is used at best in a
*synchronous* protocol, not in a UART. More likely synchronous I/O is
done by DMA, not in code. But which nowadays hardware is capable of
synchronous serial transmission at all?

DoDi


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

José Mejuto
Hello Lazarus-List,

Thursday, February 17, 2011, 12:25:26 AM, you wrote:

>> AFAIK the FIFO is in the receive, in sent the FIFOs are filled but
>> function does not return until the hardware sends the last byte.

HPD> I'm not sure, but IMO a FIFO for outgoing data is used at best in a
HPD> *synchronous* protocol, not in a UART. More likely synchronous I/O is
HPD> done by DMA, not in code. But which nowadays hardware is capable of
HPD> synchronous serial transmission at all?

Now you know why PCSpeaker has been removed from Windows ;) 8250 chip
which carries RS232 has been removed and it also controls the
PCSpeaker. The 8250 is interrupt driven without FIFO and the 16550A is
interrupt driven and FIFO 16 bytes.
16550A and alike are no more integrated in PC motherboards because it
is the last real IRQ and port mapped device so it overcomplex
circuitry and OS to handle it. Anyway the APIs are based in the same
concept as when it was interrupt and I/O driven (around 3 years ago).

Of course the 16550A chip was not included since a lot of years, but
the circuitry in replacement was compatible in operation mode
(IRQs / I/O).

--
Best regards,
 José


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Bo Berglund
In reply to this post by Michael Schnell
On Wed, 16 Feb 2011 10:53:32 +0100, Michael Schnell
<[hidden email]> wrote:

>On 02/15/2011 01:49 AM, Paulo Costa wrote:
>>
>> The SdpoSerial component creates a thread to listen and generates an
>> event when bytes are received. So reading can be non-blocking.
>> The write methods are direct calls to the Synaser ones and they are
>> blocking.
>>
>So I feel that SdpoSerial needs a buffer to hold the data read from the
>interface until the main thread is inclined to bother. The size of this
>buffer should somehow be configurable.
>

In my application I use an object that encapsulates the communication
with the instrument. I create an SdpoSerial object to do the actual
serial comm, but I also had to create a TCommBuffer class to handle
the data *from* the RS232 receive. So in the receive event of the
SdpoSerial I just dump incoming characters into this TCommBuffer
object, which keeps track of write and read indices.

The data are stuffed into a dynamic byte array, which is enlarged on
demand (but in large chunks so the overhead of expansion will not be
noticed).
I did this for receive in order to handle the "strange" protocol used
by the instrumentation.

Maybe I should expand this and use the same for transmit also, such
that the main application merely adds data to the buffer and a thread
sends the data onwards into the blocking WriteData method?
That way I can monitor the difference of the read and write indices as
a "remaining to be sent" value in a wait loop in my main application.

I might even let the thread create the SdpoSerial object so it is only
touched by the thread itself...
But I don't know what would happen because the SdpoSerial object has
internal threads of its own, I believe.

--

Bo Berglund
Developer in Sweden


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
On 02/17/2011 12:58 PM, Bo Berglund wrote:
> I might even let the thread create the SdpoSerial object so it is only
> touched by the thread itself...
If you create an object in a thread that does not mean that it's code
will run in that thread.
> But I don't know what would happen because the SdpoSerial object has
> internal threads of its own, I believe.
So maybe it might not work at all or crash when created in a thread.

-Michael


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Bo Berglund
In reply to this post by José Mejuto
On Wed, 16 Feb 2011 12:07:28 +0100, José Mejuto <[hidden email]>
wrote:

>Hello Lazarus-List,
>
>Wednesday, February 16, 2011, 10:59:02 AM, you wrote:
>
>MS> On 02/15/2011 10:45 AM, Bo Berglund wrote:
>>> Does that mean that until all of the bytes have left the serial port
>>> the Write does not return to my program? This is exactly why I asked
>>> about the buffers...
>MS> Regarding sending, there still is a FiFo (usually some 4..64  Bytes ) in
>MS> the serial hardware (hundreds or thousands of bytes) and a software Fifo
>MS> in the OS driver. So the software can't tell when something has left the
>MS> serial port. For such deeply embedded issues  the specifics of the
>MS> driver and hardware needs to be considered
>
>AFAIK the FIFO is in the receive, in sent the FIFOs are filled but
>function does not return until the hardware sends the last byte.
>
At least in the embedded world UART:s have *both* a transmit and
receive FIFO. You usually are able to set the interrupt trigger such
that it fires when there are a certain number of bytes in the Rx FIFO
and when it reduces below a certain number of bytes in the transmit
FIFO. This way one can fill the FIFO with outgoing data and then go
away doing something else until the iterrupt fires at which time the
interrupt routine will move available data from the buffer to the
FIFO.

--

Bo Berglund
Developer in Sweden


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

José Mejuto
In reply to this post by José Mejuto
Hello Hans-Peter,

Thursday, February 17, 2011, 12:28:08 PM, you wrote:

>> the 16550A is interrupt driven and FIFO 16 bytes.
HPD> And the FIFO resides in the receiver, the transmitter doesn't have a
HPD> FIFO. At least it was so, 20 years ago...

Both have FIFOs, in fact the FIFO was implemented for the transmitter
(sender) to achive a speed higher than 38400 bps and before the chip
goes out a receive FIFO was added also.

Quote: --------------------

RBR : Receiver buffer register (RO) The RBR, receiver buffer register
contains the byte received if no FIFO is used, or the oldest unread
byte with FIFO's. If FIFO buffering is used, each new read action of
the register will return the next byte, until no more bytes are
present. Bit 0 in the LSR line status register can be used to check if
all received bytes have been read. This bit wil change to zero if no
more bytes are present.

THR : Transmitter holding register (WO) The THR, transmitter holding
register is used to buffer outgoing characters. If no FIFO buffering
is used, only one character can be stored. Otherwise the amount of
characters depends on the type of UART. Bit 5 in the LSR, line status
register can be used to check if new information must be written to
THR. The value 1 indicates that the register is empty. If FIFO
buffering is used, more than one character can be written to the
transmitter holding register when the bit signals an empty state.
There is no indication of the amount of bytes currently present in the
transmitter FIFO.
--------------------------

You can get further information from http://www.lammertbies.nl/comm/info/serial-uart.html

--
Best regards,
 José


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

José Mejuto
In reply to this post by Bo Berglund
Hello Lazarus-List,

Thursday, February 17, 2011, 5:46:08 PM, you wrote:

>>AFAIK the FIFO is in the receive, in sent the FIFOs are filled but
>>function does not return until the hardware sends the last byte.
BB> At least in the embedded world UART:s have *both* a transmit and
BB> receive FIFO. You usually are able to set the interrupt trigger such
BB> that it fires when there are a certain number of bytes in the Rx FIFO
BB> and when it reduces below a certain number of bytes in the transmit
BB> FIFO. This way one can fill the FIFO with outgoing data and then go
BB> away doing something else until the iterrupt fires at which time the
BB> interrupt routine will move available data from the buffer to the
BB> FIFO.

Right, I do not correctly write the idea. I was trying to say that the
FIFO in acts like a buffer that you can forget about it (until no
receive event/interrupt) is received, while in the send FIFO the OS
waits for a full empty/transmission before return from call.

Are we completly off-topic ?

--
Best regards,
 José


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Bo Berglund
On Thu, 17 Feb 2011 21:18:54 +0100, José Mejuto <[hidden email]>
wrote:


>Are we completly off-topic ?

Maybe a tad.... ;-)
But it is about handling serial data in the best way for the
application, though.


--
Bo Berglund
Developer in Sweden


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Hans-Peter Diettrich
In reply to this post by José Mejuto
José Mejuto schrieb:

Thanks for your FIFO research :-)

> Right, I do not correctly write the idea. I was trying to say that the
> FIFO in acts like a buffer that you can forget about it (until no
> receive event/interrupt) is received, while in the send FIFO the OS
> waits for a full empty/transmission before return from call.
>
> Are we completly off-topic ?

Quite a bit. Applications don't have to care about such hardware
details, because the drivers of most systems will buffer all I/O, and
will allow for both synchronous and asynchronous I/O.

DoDi


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
In reply to this post by Bo Berglund
On 02/17/2011 05:46 PM, Bo Berglund wrote:
>
> At least in the embedded world UART:s have *both* a transmit and
> receive FIFO.
Same with PCs. And AFAIK, the OS-driver adds a huge Fifo buffer in both
directions to allow for the quite latency userland software to work as
good as possible without degrading the transmission performance.

-Michael

--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
In reply to this post by Hans-Peter Diettrich
On 02/18/2011 03:14 AM, Hans-Peter Diettrich wrote:
>
> Quite a bit. Applications don't have to care about such hardware details,

Yep and this is why user land software can't (and should not) detect the
point in time the data it issued towards the serial port actually has
left the box. These systems (Windows, Linux, Mac) are not supposed to
support hard realtime after all.

-Michael

--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Hans-Peter Diettrich
Michael Schnell schrieb:
> On 02/18/2011 03:14 AM, Hans-Peter Diettrich wrote:
>>
>> Quite a bit. Applications don't have to care about such hardware details,
>
> Yep and this is why user land software can't (and should not) detect the
> point in time the data it issued towards the serial port actually has
> left the box. These systems (Windows, Linux, Mac) are not supposed to
> support hard realtime after all.

There exists no guarantee for undisturbed transmission. Secure software
must be prepared to re-transmit the data, in its protocol, regardless of
realtime requirements. It's perfectly sufficient when the software is
informed that the next data packet can be supplied for transmission.

DoDi


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Bo Berglund
In reply to this post by Michael Schnell
On Fri, 18 Feb 2011 09:11:14 +0100, Michael Schnell
<[hidden email]> wrote:

>On 02/17/2011 05:46 PM, Bo Berglund wrote:
>>
>> At least in the embedded world UART:s have *both* a transmit and
>> receive FIFO.
>Same with PCs. And AFAIK, the OS-driver adds a huge Fifo buffer in both
>directions to allow for the quite latency userland software to work as
>good as possible without degrading the transmission performance.
>

But then we have the problem I have already detected when test running
my SdpoSerial based application:
If I set it up to send a large chunk of data and put a breakpoint at
the WriteData call so I can time the process when I click step-over I
see this:
Data size 100.000 bytes
Baudrate: 38400
Estimated shortest time for transfer: 26 s
Time to return from call: about 30 s

So this means that even if the OS has a "huge" buffer for outgoing
data it fails to improve the call time of the blocking call to
WriteData.
My initial question was about adjustment of the size of the transmuít
buffer such that I could ensure safe transfer of the data and snappy
response in my application.
As it is now the GUI will die when the transfer is made.
I have to somehow adapt to this fact and add a buffer layer myself in
a thread that feeds data to the WriteData method.

--

Bo Berglund
Developer in Sweden


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
In reply to this post by Hans-Peter Diettrich
On 02/18/2011 12:37 PM, Hans-Peter Diettrich wrote:
>  It's perfectly sufficient when the software is informed that the next
> data packet can be supplied for transmission.
>
IMHO the decent way for user land software is to use a timeout that is
_a_lot_ greater than the time the hardware needs to toggle out the
serial bits plus to receive biggest possible the answer and resend the
block if no answer is detected up till then. This is not hard realtime.
The user land software can at any time be delayed for multiple seconds
by other proceedings on the PC.

Of course a decent protocol should use a running block counter so that
the receiver can detect a block it received a second time.

-Michael

--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
In reply to this post by Bo Berglund
On 02/18/2011 02:43 PM, Bo Berglund wrote:
> I have to somehow adapt to this fact and add a buffer layer myself in
> a thread that feeds data to the WriteData method.
That is why I suggested that the best way to go is porting the
appropriate parts of AsyncPro, that perfectly does this.

Some time a go I did a quite in-depth research on this issue, yielding
this result, and I of course would have already done it, unless the
project would have been delayed for other reasons.

-Michael

--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
On 02/18/2011 04:05 PM, Michael Schnell wrote:
> porting the appropriate parts of AsyncPro, that perfectly does this.
BTW be warned: The necessary Event-mechanism (e.g
Apllication.QueuAsyncCall) does not work correctly with the current svn
Lazarus version and "gtk2" (maybe it does work with the "Windows" and/or
the "qt" Widget Type)

This might hit you when using threads for data transfer and trying to do
it in a "decent" way (without polling loops).

-Michael

--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

José Mejuto
In reply to this post by José Mejuto
Hello Lazarus-List,

Friday, February 18, 2011, 12:37:53 PM, you wrote:

>> Both have FIFOs, in fact the FIFO was implemented for the transmitter
>> (sender) to achive a speed higher than 38400 bps and before the chip
>> goes out a receive FIFO was added also.
HPD> Correction: the FIFO was implemented first for the receiver, because
HPD> lost bytes can not be recovered, and not all lines or senders allow for
HPD> hardware handshake (RTS/CTS) - that was a big problem with the old
HPD> devices. The baudrate is generated by the device, so that sending never
HPD> suffered from timing problems. [In synchronous protocol the device
HPD> inserts sync codes automatically, when no new data is supplied in time]

Well, maybe we can talk for hours about this :) as I can not show you
a document about this topic, but I'll show you my background about why
transmission buffer FIFO was more important in the beginning.
In the past times most communications in serial devices was
synchronous, so in the receiver part due the possible errors in
transferences and, of course, buffer overruns some protocols can have
some simple error correction techniques. The problem appears when
serial communications want to be far away from 38400 bps as the time
needed to raise the interrupt and just put a new byte in the buffer (1
byte buffer) is more than the character frame time which in fact
raises a buffer underrun error and a serial comunication break (break,
not error). What a change in less than 20 years! :)

--
Best regards,
 José


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

José Mejuto
In reply to this post by Michael Schnell
Hello Lazarus-List,

Friday, February 18, 2011, 4:16:31 PM, you wrote:

MS> BTW be warned: The necessary Event-mechanism (e.g
MS> Apllication.QueuAsyncCall) does not work correctly with the current svn
MS> Lazarus version and "gtk2" (maybe it does work with the "Windows" and/or
MS> the "qt" Widget Type)

You can use Syncronize, the syncronized procedure raises the event in
your code.

--
Best regards,
 José


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
Reply | Threaded
Open this post in threaded view
|

Re: [Lazarus] SdpoSerial Tx and Rx buffers?

Michael Schnell
On 02/18/2011 04:56 PM, José Mejuto wrote:
> You can use Syncronize, the syncronized procedure raises the event in
> your code.
>
Unfortunately, with these Widget sets, TThread.Synchronize does not
work, either :( (see FPC bug tracker)

Application.QueueAsyncCall is more suitable, as it
  1) allows to transfer the data block to the main thread in a single
step and
  2) does not block the thread while it waits for handling by the main
thread (Which is done in "TThread.Synchronize" (but not in the
Delphi-only "TThread.Queue").

-Michael


--
_______________________________________________
Lazarus mailing list
[hidden email]
http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus
123