Problems with support of file locks for ARC (maybe for other targets too)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Problems with support of file locks for ARC (maybe for other targets too)

Yuriy Kolerov
Hi,

I have some questions about uClibc and support of ARC targets.

1. uClibc does not support OFD locks (https://lwn.net/Articles/586904/). fcntl.h headers for all targets are not synchronized with Linux source tree and does not contain these defines:

------------------------ 8< ------------------------
#define F_OFD_GETLK 36
#define F_OFD_SETLK 37
#define F_OFD_SETLKW 38
------------------------ 8< ------------------------

Also appropriate changes must be done in "libc/sysdeps/linux/common/__syscall_fcntl.c". I am not sure how to do it in the proper manner because it involves a lot of duplication for each target...

2. uClibc uses fcntl64 instead of fcntl by default:
http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/linux/common/__syscall_fcntl.c

Also if you want to use file locks you need to pass flock or flock64 structure to fcntl which are defined in fcntl.h for each target. E.g. for ARC:
http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/linux/arc/bits/fcntl.h

So if fcntl64 is used by default then flock must be identical to flock64 in this case. That is why flock and flock64 are defined this way:

------------------------ 8< ------------------------
struct flock
  {
    short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK. */
    short int l_whence; /* Where `l_start' is relative to (like `lseek').  */
#ifndef __USE_FILE_OFFSET64
    __off_t l_start; /* Offset where the lock begins.  */
    __off_t l_len; /* Size of the locked area; zero means until EOF.  */
#else
    __off64_t l_start; /* Offset where the lock begins.  */
    __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
#endif
    __pid_t l_pid; /* Process holding the lock.  */
  };

#ifdef __USE_LARGEFILE64
struct flock64
  {
    short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK. */
    short int l_whence; /* Where `l_start' is relative to (like `lseek').  */
    __off64_t l_start; /* Offset where the lock begins.  */
    __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
    __pid_t l_pid; /* Process holding the lock.  */
  };
#endif
------------------------ 8< ------------------------

Macros __USE_FILE_OFFSET64 and __USE_LARGEFILE64 are defined in features.h when the compiler have these defines:

------------------------ 8< ------------------------
#define _FILE_OFFSET_BITS 64
#define _LARGEFILE64_SOURCE
------------------------ 8< ------------------------

So if you want to use locks you should always define those macros. Moreover the whole uClibc and every application must be compiled with those macros. Here is my question. Is it acceptable just to compile toolchain itself using CFLAGS_FOR_TARGET with those macros and don't define them manually in each application?

Regards,
Yuriy Kolerov

_______________________________________________
uClibc mailing list
[hidden email]
http://lists.busybox.net/mailman/listinfo/uclibc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problems with support of file locks for ARC (maybe for other targets too)

Vineet Gupta-5
Hi Yuriy,

On 08/09/2016 10:06 AM, Yuriy Kolerov wrote:
> 1. uClibc does not support OFD locks (https://lwn.net/Articles/586904/). fcntl.h headers for all targets are not synchronized with Linux source tree and does not contain these defines:
>
> ------------------------ 8< ------------------------
> #define F_OFD_GETLK 36
> #define F_OFD_SETLK 37
> #define F_OFD_SETLKW 38
> ------------------------ 8< ------------------------
>
> Also appropriate changes must be done in "libc/sysdeps/linux/common/__syscall_fcntl.c". I am not sure how to do it in the proper manner because it involves a lot of duplication for each target...

This is common generic ABI and applicable only to select few / newer arches.

>
> 2. uClibc uses fcntl64 instead of fcntl by default:
> http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/linux/common/__syscall_fcntl.c

FWIW - linux/common/* is the common generic syscall ABI that was added to uClibc
when ARC and metag ports were added to linux kernel. The idea is to use
asm-generic kernel headers and for syscalls uses min number of syscalls as
possible (so xxx64 only would be exposed by uapi headers. For this case
specifically, only __NR_fcntl64 will be defied in the kernel.

> Also if you want to use file locks you need to pass flock or flock64 structure to fcntl which are defined in fcntl.h for each target. E.g. for ARC:
> http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/linux/arc/bits/fcntl.h
>
> So if fcntl64 is used by default then flock must be identical to flock64 in this case. That is why flock and flock64 are defined this way:

Note that given the common generic syscall ABI - userspace doesn't need to use
struct flock64 with flock64 syscall. However the userspace struct flock needs to
match with kernel's struct flock64. See what we do for stat syscall family
(libc/sysdeps/linux/common-generic/bits/stat.h)

> ------------------------ 8< ------------------------
> struct flock
>   {
>     short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK. */
>     short int l_whence; /* Where `l_start' is relative to (like `lseek').  */
> #ifndef __USE_FILE_OFFSET64
>     __off_t l_start; /* Offset where the lock begins.  */
>     __off_t l_len; /* Size of the locked area; zero means until EOF.  */
> #else
>     __off64_t l_start; /* Offset where the lock begins.  */
>     __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
> #endif
>     __pid_t l_pid; /* Process holding the lock.  */
>   };
>
> #ifdef __USE_LARGEFILE64
> struct flock64
>   {
>     short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK. */
>     short int l_whence; /* Where `l_start' is relative to (like `lseek').  */
>     __off64_t l_start; /* Offset where the lock begins.  */
>     __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
>     __pid_t l_pid; /* Process holding the lock.  */
>   };
> #endif
> ------------------------ 8< ------------------------
>
> Macros __USE_FILE_OFFSET64 and __USE_LARGEFILE64 are defined in features.h when the compiler have these defines:
>
> ------------------------ 8< ------------------------
> #define _FILE_OFFSET_BITS 64
> #define _LARGEFILE64_SOURCE
> ------------------------ 8< ------------------------
>
> So if you want to use locks you should always define those macros.

Perhaps you know this already: these macros are part of Large File support (LFS).
Traditionally __LARGEFILE64_SOURCE means the 64 based API in explicitly used in
code, while _FILE_OFFSET_BITS=64 means compiler magically converts regular 32 bit
API into 64 bit - under the hood with preprocessor magic.

>  Moreover the whole uClibc and every application must be compiled with those macros. Here is my question. Is it acceptable just to compile toolchain itself using CFLAGS_FOR_TARGET with those macros and don’t define them manually in each application?

If you enable LFS support in build system (uClibc or buildroot) these macros will
be automatically defined - no point or need to pass them as CFLAGS etc - infact
that would be wrong IMO.

So you really don't need LFS to be able to use locks - but need to make sure that
for common generic ABI - user struct flock pairs correctly with kernel struct
flock64 with or w/o LFS.

-Vineet
_______________________________________________
uClibc mailing list
[hidden email]
http://lists.busybox.net/mailman/listinfo/uclibc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Problems with support of file locks for ARC (maybe for other targets too)

Yuriy Kolerov
Hi Veneet,

> -----Original Message-----
> From: Vineet Gupta
> Sent: Wednesday, August 10, 2016 4:01 AM
> To: Yuriy Kolerov <[hidden email]>; [hidden email]
> Cc: Waldemar Brodkorb <[hidden email]>; Alexey Brodkin
> <[hidden email]>; Anton Kolesov <[hidden email]>
> Subject: Re: Problems with support of file locks for ARC (maybe for other
> targets too)
>
> Hi Yuriy,
>
> On 08/09/2016 10:06 AM, Yuriy Kolerov wrote:
> > 1. uClibc does not support OFD locks (https://lwn.net/Articles/586904/).
> fcntl.h headers for all targets are not synchronized with Linux source tree and
> does not contain these defines:
> >
> > ------------------------ 8< ------------------------ #define
> > F_OFD_GETLK 36 #define F_OFD_SETLK 37 #define F_OFD_SETLKW 38
> > ------------------------ 8< ------------------------
> >
> > Also appropriate changes must be done in
> "libc/sysdeps/linux/common/__syscall_fcntl.c". I am not sure how to do it in
> the proper manner because it involves a lot of duplication for each target...
>
> This is common generic ABI and applicable only to select few / newer arches.
>
> >
> > 2. uClibc uses fcntl64 instead of fcntl by default:
> > http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/lin
> > ux/common/__syscall_fcntl.c
>
> FWIW - linux/common/* is the common generic syscall ABI that was added to
> uClibc when ARC and metag ports were added to linux kernel. The idea is to
> use asm-generic kernel headers and for syscalls uses min number of syscalls
> as possible (so xxx64 only would be exposed by uapi headers. For this case
> specifically, only __NR_fcntl64 will be defied in the kernel.
>
> > Also if you want to use file locks you need to pass flock or flock64 structure
> to fcntl which are defined in fcntl.h for each target. E.g. for ARC:
> > http://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/tree/libc/sysdeps/lin
> > ux/arc/bits/fcntl.h
> >
> > So if fcntl64 is used by default then flock must be identical to flock64 in this
> case. That is why flock and flock64 are defined this way:
>
> Note that given the common generic syscall ABI - userspace doesn't need to
> use struct flock64 with flock64 syscall. However the userspace struct flock
> needs to match with kernel's struct flock64. See what we do for stat syscall
> family
> (libc/sysdeps/linux/common-generic/bits/stat.h)
>
> > ------------------------ 8< ------------------------ struct flock
> >   {
> >     short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK.
> */
> >     short int l_whence; /* Where `l_start' is relative to (like `lseek').
> */
> > #ifndef __USE_FILE_OFFSET64
> >     __off_t l_start; /* Offset where the lock begins.  */
> >     __off_t l_len; /* Size of the locked area; zero means until EOF.  */
> > #else
> >     __off64_t l_start; /* Offset where the lock begins.  */
> >     __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
> > #endif
> >     __pid_t l_pid; /* Process holding the lock.  */
> >   };
> >
> > #ifdef __USE_LARGEFILE64
> > struct flock64
> >   {
> >     short int l_type; /* Type of lock: F_RDLCK, F_WRLCK, or F_UNLCK.
> */
> >     short int l_whence; /* Where `l_start' is relative to (like `lseek').
> */
> >     __off64_t l_start; /* Offset where the lock begins.  */
> >     __off64_t l_len; /* Size of the locked area; zero means until EOF.  */
> >     __pid_t l_pid; /* Process holding the lock.  */
> >   };
> > #endif
> > ------------------------ 8< ------------------------
> >
> > Macros __USE_FILE_OFFSET64 and __USE_LARGEFILE64 are defined in
> features.h when the compiler have these defines:
> >
> > ------------------------ 8< ------------------------ #define
> > _FILE_OFFSET_BITS 64 #define _LARGEFILE64_SOURCE
> > ------------------------ 8< ------------------------
> >
> > So if you want to use locks you should always define those macros.
>
> Perhaps you know this already: these macros are part of Large File support
> (LFS).
> Traditionally __LARGEFILE64_SOURCE means the 64 based API in explicitly
> used in code, while _FILE_OFFSET_BITS=64 means compiler magically
> converts regular 32 bit API into 64 bit - under the hood with preprocessor
> magic.
>
> >  Moreover the whole uClibc and every application must be compiled with
> those macros. Here is my question. Is it acceptable just to compile toolchain
> itself using CFLAGS_FOR_TARGET with those macros and don’t define them
> manually in each application?
>
> If you enable LFS support in build system (uClibc or buildroot) these macros
> will be automatically defined - no point or need to pass them as CFLAGS etc -
> infact that would be wrong IMO.

If you enable __UCLIBC_HAS_LFS__ while configuring uClibc then _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 will not be defined automatically. E.g. Buildroot has this generic line in Makefile for packages:

TARGET_CPPFLAGS += -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

These defines are not placed to config.h or somewhere else. They must be defined each time you compile anything. uClibc may be built by Buildroot with these flags by default (actually it is not true at least for ARC - Buildroot must be fixed but it is not a subject of this discussion). However you must define these macros manually when building every application later.

I don’t understand this thing - is it possible to compile uClibc with full support of LFS (see macros above) and don’t use this feature in applications and everything will be ok? And is it possible to compile uClibc without _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 and compile applications with them and everything will be ok? Right now when we build toolchain for ARC using Buildroot uClibc is compiled without them and e.g. BusyBox is compiled with -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 ...

> So you really don't need LFS to be able to use locks - but need to make sure
> that for common generic ABI - user struct flock pairs correctly with kernel
> struct
> flock64 with or w/o LFS.
>
> -Vineet
_______________________________________________
uClibc mailing list
[hidden email]
http://lists.busybox.net/mailman/listinfo/uclibc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problems with support of file locks for ARC (maybe for other targets too)

Vineet Gupta-5
On 08/11/2016 09:12 AM, Yuriy Kolerov wrote:

>
>>>  Moreover the whole uClibc and every application must be compiled with
>> those macros. Here is my question. Is it acceptable just to compile toolchain
>> itself using CFLAGS_FOR_TARGET with those macros and don’t define them
>> manually in each application?
>>
>> If you enable LFS support in build system (uClibc or buildroot) these macros
>> will be automatically defined - no point or need to pass them as CFLAGS etc -
>> infact that would be wrong IMO.
> If you enable __UCLIBC_HAS_LFS__ while configuring uClibc then _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 will not be defined automatically.

What was I smoking when replying ? You are correct - these macros have to be added
to each application's makefile or at top level as target C flags.

> E.g. Buildroot has this generic line in Makefile for packages:
>
> TARGET_CPPFLAGS += -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

Actually BR dropped support for disabling LFS in 2015 - before that the line above
used to be guarded by BR2_LARGEFILE.

> These defines are not placed to config.h or somewhere else. They must be defined each time you compile anything. uClibc may be built by Buildroot with these flags by default (actually it is not true at least for ARC - Buildroot must be fixed but it is not a subject of this discussion). However you must define these macros manually when building every application later.

Right, I stand corrected.

> I don’t understand this thing - is it possible to compile uClibc with full support of LFS (see macros above) and don’t use this feature in applications and everything will be ok?

Indeed this is the case - if you don't build with LFS, xxx64 won't even be
available to begin with.

> And is it possible to compile uClibc without _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 and compile applications with them and everything will be ok?

This mostly likely won't work as those defines will not be present in headers to
use in application.

> Right now when we build toolchain for ARC using Buildroot uClibc is compiled without them and e.g. BusyBox is compiled with -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 ...

Buildroot now assumes LFS to be default (and you can't switch it off) so
everything including uClibc / Busybox has LFS enabled already.

>
>> So you really don't need LFS to be able to use locks - but need to make sure
>> that for common generic ABI - user struct flock pairs correctly with kernel
>> struct
>> flock64 with or w/o LFS.

But this point remains valid - for our ABI - kernel uses struct flock64 and this
needs to be binary compatible with uClibc struct flock and struct flock64 (LFS only)

Read the comment in libc/sysdeps/linux/common-generic/bits/stat.h

-Vineet
_______________________________________________
uClibc mailing list
[hidden email]
http://lists.busybox.net/mailman/listinfo/uclibc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Problems with support of file locks for ARC (maybe for other targets too)

Yuriy Kolerov


> -----Original Message-----
> From: Vineet Gupta
> Sent: Friday, August 12, 2016 2:39 AM
> To: Yuriy Kolerov <[hidden email]>; [hidden email]
> Cc: Waldemar Brodkorb <[hidden email]>; Alexey Brodkin
> <[hidden email]>; Anton Kolesov <[hidden email]>;
> arcml <[hidden email]>
> Subject: Re: Problems with support of file locks for ARC (maybe for other
> targets too)
>
> On 08/11/2016 09:12 AM, Yuriy Kolerov wrote:
> >
> >>>  Moreover the whole uClibc and every application must be compiled
> >>> with
> >> those macros. Here is my question. Is it acceptable just to compile
> >> toolchain itself using CFLAGS_FOR_TARGET with those macros and don’t
> >> define them manually in each application?
> >>
> >> If you enable LFS support in build system (uClibc or buildroot) these
> >> macros will be automatically defined - no point or need to pass them
> >> as CFLAGS etc - infact that would be wrong IMO.
> > If you enable __UCLIBC_HAS_LFS__ while configuring uClibc then
> _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS=64 will not be defined
> automatically.
>
> What was I smoking when replying ? You are correct - these macros have to
> be added to each application's makefile or at top level as target C flags.
>
> > E.g. Buildroot has this generic line in Makefile for packages:
> >
> > TARGET_CPPFLAGS += -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
> > -D_FILE_OFFSET_BITS=64
>
> Actually BR dropped support for disabling LFS in 2015 - before that the line
> above used to be guarded by BR2_LARGEFILE.
>
> > These defines are not placed to config.h or somewhere else. They must be
> defined each time you compile anything. uClibc may be built by Buildroot
> with these flags by default (actually it is not true at least for ARC - Buildroot
> must be fixed but it is not a subject of this discussion). However you must
> define these macros manually when building every application later.
>
> Right, I stand corrected.
>
> > I don’t understand this thing - is it possible to compile uClibc with full
> support of LFS (see macros above) and don’t use this feature in applications
> and everything will be ok?
>
> Indeed this is the case - if you don't build with LFS, xxx64 won't even be
> available to begin with.
>
> > And is it possible to compile uClibc without _LARGEFILE64_SOURCE and
> _FILE_OFFSET_BITS=64 and compile applications with them and everything
> will be ok?
>
> This mostly likely won't work as those defines will not be present in headers
> to use in application.

But actually it works... As for now uClibc in Buildroot is built without -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 because uClibc does not use global TARGET_CFLAGS. And all other applications (e.g. BusyBox) use those flags. Again - all those defines are presented in features.h and must be activating just by passing those -D options. I'll try to push a patch to Buildroot to force using LFS flags but what frustrates me is that everything works somehow together.

> > Right now when we build toolchain for ARC using Buildroot uClibc is
> compiled without them and e.g. BusyBox is compiled with -
> D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64
> ...
>
> Buildroot now assumes LFS to be default (and you can't switch it off) so
> everything including uClibc / Busybox has LFS enabled already.
>
> >
> >> So you really don't need LFS to be able to use locks - but need to
> >> make sure that for common generic ABI - user struct flock pairs
> >> correctly with kernel struct
> >> flock64 with or w/o LFS.
>
> But this point remains valid - for our ABI - kernel uses struct flock64 and this
> needs to be binary compatible with uClibc struct flock and struct flock64 (LFS
> only)
>
> Read the comment in libc/sysdeps/linux/common-generic/bits/stat.h
>
> -Vineet
_______________________________________________
uClibc mailing list
[hidden email]
http://lists.busybox.net/mailman/listinfo/uclibc
Loading...