afsd: Avoid fscanf overflows when paring cacheinfo clang-14 is producing the following diagnostic: afsd.c:581:44: error: 'fscanf' may overflow; destination buffer in argument 3 has size 1024, but the corresponding specifier may require size 1025 [-Werror,-Wfortify-source] fscanf(cachefd, "%1024[^:]:%1024[^:]:%d", tCacheMountDir, fscanf is being used to parse the contents of a file and the buffer sizes are hardcoded. Simply increase the size of the 2 buffers by 1. The diagnostic warning is changed to an error when configured with --enable-checking. Change-Id: Iefbc4e87242232531a266e876fe779476b42fb62 Reviewed-on: https://gerrit.openafs.org/14958 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
macos: Add support for Apple Silicon This commit introduces the new set of changes / files required to successfully build the OpenAFS source code on Apple M1 (macOS 11+). Notice that kexts on Apple silicon must support the arm64e architecture, since they run in kernel space and must support the same architecture and restrictions as other kernel code. On the other hand, the utilities that run on user space, like vos, fs, and others, have to be built for arm64 (currently). Change-Id: Ib005215ae9bfb0b9dda94b2e194f05c0ce92fa1e Reviewed-on: https://gerrit.openafs.org/14746 Reviewed-by: Andrew Deason <adeason@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: Andrew Deason <adeason@sinenomine.net>
IRIX: Remove pre-65 code Commit d1923139 (irix kill efs and start pruning pre-65) removed all files that defined AFS_SGI64_ENV and earlier, but didn't remove that code that depended on those defines. In addition, there has been code in the tree that checks for AFS_SGI53_ENV since OpenAFS 1.0, but nothing has ever defined Irix 5.3 support in OpenAFS. Remove all of this obsolete code. Change all references to AFS_SGIXX_ENV to AFS_SGI_ENV, and assume AFS_SGI_ENV is defined in all IRIX dirs. Consolidate some of the resulting ifdef logic appropriately. Change-Id: I9dd426296e04801980b805a5e195063762b23189 Reviewed-on: https://gerrit.openafs.org/14230 Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: Benjamin Kaduk <kaduk@mit.edu>
Cleanup AFS_*LINUX_ENV usage Commit 6329a523f6305541871bc3d1694065e7b17abe77 changed all occurrences of AFS_*LINUXnn_ENV to AFS_*LINUX_ENV, but did not perform any refactoring of the use of these variables. This commit completes the task by refactoring the preprocessor conditionals that involved removing dead code or collapsing statements. The updates should have no functional changes. Change-Id: I10fbefb7c862de2ec35ab77cafa2391e6e59f388 Reviewed-on: https://gerrit.openafs.org/14388 Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: BuildBot <buildbot@rampaginggeek.com>
Change AFS*_LINUXnn_ENV to AFS*_LINUX_ENV The minimum Linux kernel that is now supported is linux-2.6.18. The Linux versioned preprocessor macros AFS_*LINUXnn_ENV are no longer needed to distinguish the different levels of Linux and can be merged into just a single set of macros. Perform a global change of _LINUX\d+_ENV to _LINUX_ENV. e.g. AFS_LINUX24_ENV -> AFS_LINUX_ENV AFS_USR_LINUX24_ENV -> AFS_USR_LINUX_ENV AFS_AMD64_LINUX20_ENV -> AFS_AMD64_LINUX_ENV Replace the multiple definitions for the versioned 'AFS*_LINUXnn_ENV' with just single non-version definitions 'AFS*_LINUX_ENV'. Apart from replacing the now-redundant #define directives and tidying up a few comments at the close of a preprocessor block to match their current form, this commit was done using a mechanical change of the variable names and did not reduce preprocessor statements that could now be combined or eliminated. Nor does this commit remove dead code. A follow-up commit (Cleanup AFS_*LINUX_ENV usage) will handle these changes. The updates should have no functional changes. Change-Id: I71e32ca9818d28f82b4f23679868d1b9a62c44bd Reviewed-on: https://gerrit.openafs.org/14387 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
afs: clarify cold and warm shutdown logic Currently, any code that wants to perform a cold shutdown must first set global afs_cold_shutdown = 1, then call afs_shutdown(void). Instead, modify afs_shutdown() to accept a single parm which specifies AFS_WARM or AFS_COLD shutdown, and to set the value of global afs_cold_shutdown based on this parm. Remove all other assignments for afs_cold_shutdown. Modify all callers of afs_shutdown() to specify AFS_WARM or AFS_COLD as needed to maintain equivalent function. This should make it much easier to tell at a glance what type of shutdown is being requested by each caller to afs_shutdown(). No functional change should be incurred by this commit. Change-Id: I921eca5b4d2659209154fbe37d575db69bf708b8 Reviewed-on: https://gerrit.openafs.org/12182 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Andrew Deason <adeason@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
afsd: remove unused variable afs_shutdown Since the original IBM code import, the variable afs_shutdown has been set but never read. Remove it from the code base. No functional change is incurred by this commit. Change-Id: Iac603465ee911fa3601010b67f3f34f22d7b0e3f Reviewed-on: https://gerrit.openafs.org/14380 Reviewed-by: Andrew Deason <adeason@sinenomine.net> Reviewed-by: Cheyenne Wills <cwills@sinenomine.net> Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
macos: delegate sock_* calls to bkg daemons As part of Apple's ongoing effort to modernize macOS, improve security and reliability, the deprecation of kernel extensions was officially announced at WWDC19. According to this announcement, Kernel programming interfaces will be deprecated as alternatives become available, and future OS releases will no longer load kernel extensions that use deprecated KPIs by default. Unfortunately, the following KPIs, extensively used by rx, are included in the list of deprecated KPIs as of macOS 10.15: - sock_receivembuf - sock_close - sock_send - sock_socket - sock_setsockopt - sock_bind To workaround this problem, delegate calls to the functions mentioned above to bkg daemons forked by afsd. Notice that the ifadd_* and ifnet_* functions are also deprecated. Fortunately, these calls can be avoided enabling AFS_USERSPACE_IP_ADDR. Thanks to Andrew Deason for his assistance (ideas, suggestions, documentation, etc). Change-Id: I916b66455bec73138c55e2764cc1146b998cb19f Reviewed-on: https://gerrit.openafs.org/14431 Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: BuildBot <buildbot@rampaginggeek.com>
afs: Return to userspace after AFS_NEW_BKG reqs Currently, for AFS_NEW_BKG, background daemons run in the context of a normal user process (afsd), in order to return to run userspace-handled background ops. For non-AFS_NEW_BKG when AFS_DAEMONOP_ENV is defined, background daemons run as kernel threads instead, and have no corresponding userspace process. On LINUX, whether or not we run as a kernel thread has some odd side-effects: at least one example of this is how open file handles (struct file) are treated when closed. When the last reference to a struct file is closed, the final free is deferred to an asynchronous call to be executed "later", in order to avoid issues with lock inversion. For kernel threads, "later" means the work is schedule on the global system work queue (schedule_work()), but for userspace processes, it is scheduled on the task work queue (task_work_add()), which is run around when the thread returns to userspace. For background daemons, we never return from the relevant syscall until we get a userspace background request (or the client is shutting down), Commit ca472e66 (LINUX: Turn on AFS_NEW_BKG) changed LINUX to use AFS_NEW_BKG background daemons, so background requests now run as a normal userspace process, and last-reference file closes are deferred. Since we may never return to userspace, this means that our file handles (used for accessing the disk cache) may never get freed, leading to an unbounded number of file handles remaining open. This can be seen by seeing the first value in /proc/sys/fs/file-nr growing without bound (possibly slowly), as accessing /afs causes background requests. Eventually the number of open files can exceed the /proc/sys/fs/file-max limit, causing further file opens to fail, causing various other problems and potentially panics. To avoid this issue, define a new userspace background op, called AFS_USPC_NOOP, which gets returned to the afsd background daemon process. When afsd sees this, it just does nothing and calls the AFSOP_BKG_HANDLER syscall again, to go into the background daemon loop again. In afs_BackgroundDaemon, we return the AFS_USPC_NOOP op whenever there are no pending background requests, or if we've run 100 background requests in a row without a break. This causes us to return to userspace periodically, preventing any such task-queued work from building up indefinitely. Do this for all platforms (currently just LINUX and DARWIN), in order to simplify the code, and possibly avoid other similar issues, since staying inside a syscall for so long while doing real work is arguably weird. Add a documentation comment block for afs_BackgroundDaemon while we're here. Thanks to mvitale@sinenomine.net for discovering the file leak. Change-Id: I1953d73b2142d4128b064f8c5f95a5858d7df131 Reviewed-on: https://gerrit.openafs.org/13984 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Remove dead code There is a perhaps-surprisingly large amount of code disabled behind directives like '#if 0', '#ifdef notdef', and '#ifdef notyet'. At best, this code is clutter, and at worst some of it is confusing/outdated, and/or confusingly nested inside other preprocessor conditionals. Sometimes this disabled code shows up when grepping the tree, and causes a nuisance when refactoring related areas of code. Get rid of all of it. If anyone ever wants this code back, it can always be restored by reverting portions of this commit. Also delete some comments that clearly refer to the disabled code, and in some cases, adjust the adjacent comments to make sense accordingly. This commit doesn't touch any files in src/external/. Change-Id: If260a41257e8d107930bd3c177eddb8ab336f0d1 Reviewed-on: https://gerrit.openafs.org/13683 Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: Benjamin Kaduk <kaduk@mit.edu>
LINUX: Turn on AFS_NEW_BKG AFS_NEW_BKG allows libafs to request the afsd background daemon processes to do certain userspace operations. This is currently only used on DARWIN for handling EXDEV file moves, but this framework can be useful on LINUX, as well. So, turn it on for LINUX. This commit does not introduce any new background operations for LINUX to actually use; we're just turning on the new framework. Future commits will introduce new background operations. Change-Id: I5d371f85b87899ce6ab2d5e520954a893679d37e Reviewed-on: https://gerrit.openafs.org/13284 Reviewed-by: Mark Vitale <mvitale@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: BuildBot <buildbot@rampaginggeek.com>
afs: Add AFS_USPC_SHUTDOWN bkg request When AFS_NEW_BKG was added, the kernel module indicated to the relevant afsd process that it's time to shutdown by returning -2. This works on DARWIN, but it's difficult to make this work on all platforms, because of the different way that platforms handle error codes from our pioctls and other AFS syscalls. Specifically, on LINUX, negative error codes are assumed to be negative errno codes, and so returning -2 from the syscall handler means we return -1 to userspace, with errno set to 2 (ENOENT). Getting this to work consistently across platforms is probably more trouble than its worth, so instead of relying on specific return codes from the syscall, just add a new background daemon operation called AFS_USPC_SHUTDOWN, which just tells the background daemon to exit. Change-Id: I00b245c8f734dc9e49d6b4268cd0f6a4f1896894 Reviewed-on: https://gerrit.openafs.org/13281 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
Remove DUX/OSF code Remove code for DUX/OSF platforms. DUX code was removed from the libafs client in commit 392dcf67 ("Complete removal of DUX client code") and the alpha_dux* param files were removed in dc4d9d64 ("afs: Remove AFS_BOZONLOCK_ENV"). This code has always been disabled since those commits, so remove any code referencing AFS_DUX*_ENV, AFS_OSF_ENV, and related symbols. Change-Id: I3787b83c80a48e53fe214fdecf9a9ac0b63d390c Reviewed-on: https://gerrit.openafs.org/13260 Reviewed-by: Michael Meffie <mmeffie@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: Benjamin Kaduk <kaduk@mit.edu>
afs: Return memcache allocation errors During cache initialization, we can fail to allocate our dcache entries for memcache. Currently when this happens, we just log a message and try to disable dcache access. However, this results in at least one code path that causes a panic anyway during startup, since afs_CacheTruncateDaemon will try to trim the cache, and afs_GetDownD will call afs_MemGetDSlot, and we cannot find the given dslot. To avoid this, change our cache initialization to return an error, instead of trying to continue without a functional dcache. This causes afs_dcacheInit to return an error in this case, and by extension afs_CacheInit and the AFSOP_CACHEINIT syscall. Also change afsd to actually detect errors from AFSOP_CACHEINIT, and to bail out when it does. Thanks to gsgatlin@ncsu.edu for reporting the relevant panic. Change-Id: Ic89ff9638201faae6c4399a2344d4da3e251d537 Reviewed-on: https://gerrit.openafs.org/13273 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Michael Meffie <mmeffie@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
afs: improve -volume-ttl error messages Change the afs call which sets the volume ttl value to return EFAULT instead of EINVAL when given an out of range value for the volume ttl parameter. This is more consistent with the other op codes, which return EFAULT when given an out of range parameter and allows the caller to distinguish between an invalid opcode and a bad parameter. Move the volume ttl range constants to afs_args.h, which is where constants related to the op codes are supposed to be defined. This makes the constants available to the caller in afsd.c as well as the implementation in afs_call.c. Update afsd to print a more sensible error message when the volume ttl set calls fails due to an out of range parameter. Change-Id: I6b3ab7d38a60464017daf06f70080a90d2a7a429 Reviewed-on: https://gerrit.openafs.org/12918 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
libafs: vldb cache timeout option (-volume-ttl) The unix cache manager caches VLDB information for read-only volumes as long as a volume callback is held for a read-only volume. The volume callback may be held as long as files in the read-only volume are being accessed. The cache manager caches VLDB information for read/write volumes as long as volume level errors (such as VMOVED) are not returned by a fileserver while accessing files within the volume. Add a new option to set the maximum amount of time VLDB information will be cached, even if a callback is still held for a read-only volume, or no volume errors have been encounted while accessing files in read/write volumes. This avoids situations where the vldb information is cached indefinitely for read-only and read/write volumes. Instead, the VL servers will be periodically probed for volume information. Change-Id: I5f2a57cdaf5cbe7b1bc0440ed6408226cc988fed Reviewed-on: https://gerrit.openafs.org/11898 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Michael Meffie <mmeffie@sinenomine.net> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
src/afsd/afsd.c: Fix misleading indentation Fixes these warnings (errors with --enable-checking) from GCC 6.2: afsd.c: In function ‘afsd_run’: afsd.c:2176:6: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation] if (enable_rxbind) ^~ afsd.c:2178:3: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’ afsd_syscall(AFSOP_ADVISEADDR, code, addrbuf, maskbuf, mtubuf); ^~~~~~~~~~~~ afsd.c:2487:5: error: this ‘if’ clause does not guard... [-Werror=misleading-indentation] if (afsd_debug) ^~ afsd.c:2490:2: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’ afsd_syscall(AFSOP_GO, 0); ^~~~~~~~~~~~ Change-Id: Ic4769046dc06bb58d61428ac08ea12a2f70743e9 Reviewed-on: https://gerrit.openafs.org/12438 Reviewed-by: Benjamin Kaduk <kaduk@mit.edu> Tested-by: BuildBot <buildbot@rampaginggeek.com>
afsd: fix afsd -help crash afsd crashes after the usage is displayed with the -help option. $ afsd -help Usage: ./afsd [-blocks <1024 byte blocks in cache>] [-files <files in cache>] ... Segmentation fault (core dumped) The backtrace shows the crash occurs when calling afsconf_Open() with an invalid pointer argument, even though afsconf_Open() is not even needed when -help is given. (gdb) bt #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32 #1 0x00007ffff726fc36 in *__GI___strdup (s=0x0) at strdup.c:42 #2 0x0000000000408383 in afsconf_Open (adir=0x0) at cellconfig.c:444 #3 0x00000000004054d5 in afsd_run () at afsd.c:1926 #4 0x0000000000407dc5 in main (argc=2, argv=0x7fffffffe348) at afsd_kernel.c:577 afsconf_Open() is called with an uninitialized pointer because commit d72df5a18e0bb8bbcbf23df3e8591072f0cdb770 changed the libcmd cmd_Dispatch() to return 0 after displaying the command usage when the -help option is specified. (That fix was needed for scripts which use the -help option to inspect command options with the -help option.) The afsd_kernel main function then incorrectly calls the afsd_run() function, even though mainproc() was not called, which sets up the afsd option variables. The afsconf_Open() is the first function we call in afsd_run(). Commit f77c078a291025d593f3170c57b6be5f257fc3e5 split afsd into afsd.c and afsd_kernel.c to support libuafs (and fuse). This split the parsing of the command line arguments and the running of the afsd command into two functions. The mainproc(), which originally did both, was split into two functions; one (still called mainproc) to check the option values given and setup/auto-tune values, and another (called afsd_run) to do the actual running of the afsd command. The afsd_parse() function was introduced as a wrapper around cmd_Dispatch() which "dispatches" mainproc. With this fix, take the opportunity to rename mainproc() to the now more accurately named CheckOptions() and change afsd_parse() to parse the command line options with cmd_Parse(), instead of abusing cmd_Dispatch(). Change the main fuction to avoid running afsd_run() when afsd_parse() returns the CMD_HELP code which indicates the -help option was given. afsd.fuse splits the command line arguments into afsd recognized options and fuse options (everything else), so only afsd recognized arguments are passed to afsd_parse(), via uafs_ParseArgs(). The -help argument is processed as part of that splitting of arguments, so afsd.fuse never passes -help as an argument to afsd_parse(). This means we to not need to check for CMD_HELP as a return value from uafs_ParseArgs(). But since this is all a bit confusing, at least check the return value in uafs_ParseArgs(). Change-Id: If510f8dc337e441c19b5e28685e2e818ff57ef5a Reviewed-on: https://gerrit.openafs.org/12360 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
afs: add afsd -inumcalc option This commit adds the afsd -inumcalc command line switch to specify the inode number calculation method in a platform neutral way. Inode numbers reported for files within the AFS filesystem are generated by the cache manager using a calculation which derives a number from a FID. Long ago, a new type of calculation was added which generates inode numbers using a MD5 message digest of the FID. The MD5 inode number calculation variant is computationally more expensive but greatly reduces the chances for inode number collisions. The MD5 calculation can be enabled on the Linux cache manager using the Linux sysctl interface. Other than the sysctl method of selecting the inode calculation type, the MD5 inode number calculation method is not specific to Linux. This change introduces a command-line option which accepts a value to indicate the calculation method, instead of a simple flag to enable MD5 inode numbers. This should allow for new inode calculation methods in the future without the need for additional afsd command-line flags. Two values are currently accepted for -inumcalc. The value of 'compat' specifies the legacy inode number calculation. The value 'md5' indicates that the new MD5 calculation is to be used. Change-Id: I0257c68ca1a32a7a4c55ca8174a4926ff78ddea4 Reviewed-on: https://gerrit.openafs.org/11855 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Benjamin Kaduk <kaduk@mit.edu>
afsd: Update list of supported flags afsd.c starts with a block comment listing the flags supported by the afsd command. As the code has evolved this list has not been kept up to date. Bring the list up to date. Some obsolete options no longer have any backing code. These are marked OBSOLETE. Some obsolete options have code that says they are now deprecated. These are marked IGNORED. Additionally fix a typo in backuptree's help text. Change-Id: I90ddf4db826c891bf1daf6959f94feee17d35f78 Reviewed-on: http://gerrit.openafs.org/11857 Tested-by: BuildBot <buildbot@rampaginggeek.com> Reviewed-by: Daria Brashear <shadow@your-file-system.com>