+++ /dev/null
-The Demand-Attach FileServer (DAFS) has resulted in many changes to how
-many things on AFS fileservers behave. The most sweeping changes are
-probably in the volume package, but significant changes have also been
-made in the SYNC protocol, the vnode package, salvaging, and a few
-miscellaneous bits in the various fileserver processes.
-
-This document serves as an overview for developers on how to deal with
-these changes, and how to use the new mechanisms. For more specific
-details, consult the relevant doxygen documentation, the code comments,
-and/or the code itself.
-
- - The salvageserver
-
-The salvageserver (or 'salvaged') is a new OpenAFS fileserver process in
-DAFS. This daemon accepts salvage requests via SALVSYNC (see below), and
-salvages a volume group by fork()ing a child, and running the normal
-salvager code (it enters vol-salvage.c by calling SalvageFileSys1).
-
-Salvages that are initiated from a request to the salvageserver (called
-'demand-salvages') occur automatically; whenever the fileserver (or
-other tool) discovers that a volume needs salvaging, it will schedule a
-salvage on the salvageserver without any intervention needed.
-
-When scheduling a salvage, the vol id should be the id for the volume
-group (the RW vol id). If the salvaging child discovers that it was
-given a non-RW vol id, it will send the salvageserver a SALVSYNC LINK
-command, and will exit. This will tell the salvageserver that whenever
-it receives a salvage request for that vol id, it should schedule a
-salvage for the corresponding RW id instead.
-
- - FSSYNC/SALVSYNC
-
-The FSSYNC and SALVSYNC protocols are the protocols used for
-interprocess communication between the various fileserver processes.
-FSSYNC is used for querying the fileserver for volume metadata,
-'checking out' volumes from the fileserver, and a few other things.
-SALVSYNC is used to schedule and query salvages in the salvageserver.
-
-FSSYNC existed prior to DAFS, but it encompasses a much larger set of
-commands with the advent of DAFS. SALVSYNC is entirely new to DAFS.
-
- -- SYNC
-
-FSSYNC and SALVSYNC are both layered on top of a protocol called SYNC.
-SYNC isn't much a protocol in itself; it just handles some boilerplate
-for the messages passed back and forth, and some error codes common to
-both FSSYNC and SALVSYNC.
-
-SYNC is layered on top of TCP/IP, though we only use it to communicate
-with the local host (usually via a unix domain socket). It does not
-handle anything like authentication, authorization, or even things like
-serialization. Although it uses network primitives for communication,
-it's only useful for communication between processes on the same
-machine, and that is all we use it for.
-
-SYNC calls are basically RPCs, but very simple. The calls are always
-synchronous, and each SYNC server can only handle one request at a time.
-Thus, it is important for SYNC server handlers to return as quickly as
-possible; hitting the network or disk to service a SYNC request should
-be avoided to the extent that such is possible.
-
-SYNC-related source files are src/vol/daemon_com.c and
-src/vol/daemon_com.h
-
- -- FSSYNC
-
- --- server
-
-The FSSYNC server runs in the fileserver; source is in
-src/vol/fssync-server.c.
-
-As mentioned above, FSSYNC handlers should finish quickly when
-servicing a request, so hitting the network or disk should be avoided.
-In particular, you absolutely cannot make a SALVSYNC call inside an
-FSSYNC handler; the SALVSYNC client wrapper routines actively prevent
-this from happening, so even if you try to do such a thing, you will not
-be allowed to. This prohibition is to prevent deadlock, since the
-salvageserver could have made the FSSYNC request that you are servicing.
-
-When a client makes a FSYNC_VOL_OFF or NEEDVOLUME request, the
-fileserver offlines the volume if necessary, and keeps track that the
-volume has been 'checked out'. A volume is left online if the checkout
-mode indicates the volume cannot change (see VVolOpLeaveOnline_r).
-
-Until the volume has been 'checked in' with the ON, LEAVE_OFFLINE, or
-DONE commands, no other program can check out the volume.
-
-Other FSSYNC commands include abilities to query volume metadata and
-stats, to force volumes to be attached or offline, and to update the
-volume group cache. See doc/arch/fssync.txt for documentation on the
-individual FSSYNC commands.
-
- --- clients
-
-FSSYNC clients are generally any OpenAFS process that runs on a
-fileserver and tries to access volumes directly. The volserver,
-salvageserver, and bosserver all qualify, as do (sometimes) some
-utilities like vol-info or vol-bless. For issuing FSSYNC commands
-directly, there is the debugging tool fssync-debug. FSSYNC client code
-is in src/vol/fssync-client.c, but it's not very interesting.
-
-Any program that wishes to directly access a volume on disk must check
-out the volume via FSSYNC (NEEDVOLUME or OFF commands), to ensure the
-volume doesn't change while the program is using it. If the program
-determines that the volume is somehow inconsistent and should be
-salvaged, it should send the FSSYNC command FORCE_ERROR with reason code
-FSYNC_SALVAGE to the fileserver, which will take care of salvaging it.
-
- -- SALVSYNC
-
-The SALVSYNC server runs in the salvageserver; code is in
-src/vol/salvsync-server.c. SALVSYNC clients are just the fileserver, the
-salvageserver run with the -client switch, and the salvageserver worker
-children. If any other process notices that a volume needs salvaging, it
-should issue a FORCE_ERROR FSSYNC command to the fileserver with the
-FSYNC_SALVAGE reason code.
-
-The SALVSYNC protocol is simpler than the FSSYNC protocol. The commands
-are basically just to create, cancel, change, and query salvages. The
-RAISEPRIO command increases the priority of a salvage job that hasn't
-started yet, so volumes that are accessed more frequently will get
-salvaged first. The LINK command is used by the salvageserver worker
-children to inform the salvageserver parent that it tried to salvage a
-readonly volume for which a read-write clone exists (in which case we
-should just schedule a salvage for the parent read-write volume).
-
-Note that canceling a salvage is just for salvages that haven't run
-yet; it only takes a salvage job off of a queue; it doesn't stop a
-salvageserver worker child in the middle of a salvage.
-
- - The volume package
-
- -- refcounts
-
-Before DAFS, the Volume struct just had one reference count, vp->nUsers.
-With DAFS, we know have the notion of an internal/lightweight reference
-count, and an external/heavyweight reference count. Lightweight refs are
-acquired with VCreateReservation_r, and released with
-VCancelReservation_r. Heavyweight refs are acquired as before, normally
-with a GetVolume or AttachVolume variant, and releasing the ref with
-VPutVolume.
-
-Lightweight references are only acquired within the volume package; a vp
-should not be given to e.g. the fileserver code with an extra
-lightweight ref. A heavyweight ref is generally acquired for a vp that
-will be given to some non-volume-package code; acquiring a heavyweight
-ref guarantees that the volume header has been loaded.
-
-Acquiring a lightweight ref just guarantees that the volume will not go
-away or suddenly become unavailable after dropping VOL_LOCK. Certain
-operations like detachment or scheduling a salvage only occur when all
-of the heavy and lightweight refs go away; see VCancelReservation_r.
-
- -- state machine
-
-Instead of having a per-volume lock, each vp always has an associated
-'state', that says what, if anything, is occurring to a volume at any
-particular time; or if the volume is attached, offline, etc. To do the
-basic equivalent of a lock -- that is, ensure that nobody else will
-change the volume when we drop VOL_LOCK -- you can put the volume in
-what is called an 'exclusive' state (see VIsExclusiveState).
-
-When a volume is in an exclusive state, no thread should modify the
-volume (or expect the vp data to stay the same), except the thread that
-put it in that state. Whenever you manipulate a volume, you should make
-sure it is not in an exclusive state; first call VCreateReservation_r to
-make sure the volume doesn't go away, and then call
-VWaitExclusiveState_r. When that returns, you are guaranteed to have a
-vp that is in a non-exclusive state, and so can me manipulated. Call
-VCancelReservation_r when done with it, to indicate you don't need it
-anymore.
-
-Look at the definition of the VolState enumeration to see all volume
-states, and a brief explanation of them.
-
- -- VLRU
-
-See: Most functions with VLRU in their name in src/vol/volume.c.
-
-The VLRU is what dictates when volumes are detached after a certain
-amount of inactivity. The design is pretty much a generational garbage
-collection mechanism. There are 5 queues that a volume can be on the
-VLRU (VLRUQueueName in volume.h). 'Candidate' volumes haven't seen
-activity in a while, and so are candidates to be detached. 'New' volumes
-have seen activity only recently; 'mid' volumes have seen activity for
-awhile, and 'old' volumes have seen activity for a long while. 'Held'
-volumes cannot be soft detached at all.
-
-Volumes are moved from new->mid->old if they have had activity recently,
-and are moved from old->mid->new->candidate if they have not had any
-activity recently. The definition of 'recently' is configurable by the
--vlruthresh fileserver parameter; see VLRU_ComputeConstants for how they
-are determined. Volumes start at 'new' on attachment, and if any
-activity occurs when a volume is on 'candidate', it's moved to 'new'
-immediately.
-
-Volumes are generally promoted/demoted and soft-detached by
-VLRU_ScannerThread, which runs every so often and moves volumes between
-VLRU queues depending on their last access time and the various
-thresholds (or soft-detaches them, in the case of the 'candidate'
-queue). Soft-detaching just means the volume is taken offline and put
-into the preattached state.
-
- --- DONT_SALVAGE
-
-The dontSalvage flag in volume headers can be set to DONT_SALVAGE to
-indicate that a volume probably doesn't need to be salvaged. Before
-DAFS, volumes were placed on an 'UpdateList' which was periodically
-scanned, and dontSalvage was set on volumes that hadn't been touched in
-a while.
-
-With DAFS and the VLRU additions, setting dontSalvage now happens when a
-volume is demoted a VLRU generation, and no separate list is kept. So if
-a volume has been idle enough to demote, and it hasn't been accessed in
-SALVAGE_INTERVAL time, dontSalvage will be set automatically by the VLRU
-scanner.
-
- -- Vnode
-
-Source files: src/vol/vnode.c, src/vol/vnode.h, src/vol/vnode_inline.h
-
-The changes to the vnode package are largely very similar to those in
-the volume package. A Vnode is put into specific states, some of which
-are exclusive and act like locks (see VnChangeState_r,
-VnIsExclusiveState). Vnodes also have refcounts, incremented and
-decremented with VnCreateReservation_r and VnCancelReservation_r like
-you would expect. I/O should be done outside of any global locks; just
-the vnode is 'locked' by being put in an exclusive state if necessary.
-
-In addition to a state, vnodes also have a count of readers. When a
-caller gets a vnode with a read lock, we of course must wait for the
-vnode to be in a nonexclusive state (VnWaitExclusive_r), then the number
-of readers is incremented (VnBeginRead_r), but the vnode is kept in a
-non-exclusive state (VN_STATE_READ).
-
-When a caller gets a vnode with a write lock, we must wait not only for
-the vnode to be in a nonexclusive state, but also for there to be no
-readers (VnWaitQuiescent_r), so we can actually change it.
-
-VnLock still exists in DAFS, but it's almost a no-op. All we do for DAFS
-in VnLock is set vnp->writer to the current thread id for a write lock,
-for some consistency checks later (read locks are actually no-ops).
-Actual mutual exclusion in DAFS is done by the vnode state machine and
-the reader count.
-
- - viced state serialization
-
-See src/viced/serialize_state.* and ShutDownAndCore in
-src/viced/viced.c
-
-Before DAFS, whenever a fileserver restarted, it lost all information
-about all clients, what callbacks they had, etc. So when a client with
-existing callbacks contacted the fileserver, all callback information
-needed to be reset, potentially causing a bunch of unnecessary traffic.
-And of course, if the client does not contact the fileserver again, it
-could not get sent callbacks it should get sent.
-
-DAFS now has the ability to save the host and CB data to a file on
-shutdown, and restore it when it starts up again. So when a fileserver
-is restarted, the host and CB information should be effectively the same
-as when it shut down. So a client may not even know if a fileserver was
-restarted.
-
-Getting this state information can be a little difficult, since the host
-package data structures aren't necessarily always consistent, even after
-H_LOCK is dropped. What we attempt to do is stop all of the background
-threads early in the shutdown process (set fs_state.mode -
-FS_MODE_SHUTDOWN), and wait for the background threads to exit (or be
-marked as 'tranquil'; see the fs_state struct) later on, before trying
-to save state. This makes it a lot less likely for anything to be
-modifying the host or CB structures by the time we try to save them.
-
- - volume group cache
-
-See: src/vol/vg_cache* and src/vol/vg_scan.c
-
-The VGC is a mechanism in DAFS to speed up volume salvages. Pre-VGC,
-whenever the salvager code salvaged an individual volume, it would need
-to read all of the volume headers on the partition, so it knows what
-volumes are in the volume group it is salvaging, so it knows what
-volumes to tell the fileserver to take offline. With demand-salvages,
-this can make salvaging take a very long time, since the time to read in
-all volume headers can take much more time than the time to actually
-salvage a single volume group.
-
-To prevent the need to scan the partition volume headers every single
-time, the fileserver maintains a cache of which volumes are in what
-volume groups. The cache is populated by scanning a partition's volume
-headers, and is started in the background upon receiving the first
-salvage request for a partition (VVGCache_scanStart_r,
-_VVGC_scan_start).
-
-After the VGC is populated, it is kept up to date with volumes being
-created and deleted via the FSSYNC VG_ADD and VG_DEL
-commands. These are called every time a volume header is created,
-removed, or changed when using the volume header wrappers in vutil.c
-(VCreateVolumeDiskHeader, VDestroyVolumeDiskHeader,
-VWriteVolumeDiskHeader). These wrappers should always be used to
-create/remove/modify vol headers, to ensure that the necessary FSSYNC
-commands are called.
-
- -- race prevention
-
-In order to prevent races between volume changes and VGC partition scans
-(that is, someone scans a header while it is being written and not yet
-valid), updates to the VGC involving adding or modifying volume headers
-should always be done under the 'partition header lock'. This is a
-per-partition lock to conceptually lock the set of volume headers on
-that partition. It is only read-held when something is writing to a
-volume header, and it is write-held for something that is scanning the
-partition for volume headers (the VGC or partition salvager). This is a
-little counterintuitive, but it is what we want. We want multiple
-headers to be written to at once, but if we are the VGC scanner, we want
-to ensure nobody else is writing when we look at a header file.
-
-Because the race described above is so rare, vol header scanners don't
-actually hold the lock unless a problem is detected. So, what they do is
-read a particular volume header without any lock, and if there is a
-problem with it, they grab a write lock on the partition vol headers,
-and try again. If it still has a problem, the header is just faulty; if
-it's okay, then we avoided the race.
-
-Note that destroying vol headers does not require any locks, since
-unlink()s are atomic and don't cause any races for us here.
-
- - partition and volume locking
-
-Previously, whenever the volserver would attach a volume or the salvager
-would salvage anything, the partition would be locked
-(VLockPartition_r). This unnecessarily serializes part of most volserver
-operations. It also makes it so only one salvage can run on a partition
-at a time, and that a volserver operation cannot occur at the same time
-as a salvage. With the addition of the VGC (previous section), the
-salvager partition lock is unnecessary on namei, since the salvager does
-not need to scan all volume headers.
-
-Instead of the rather heavyweight partition lock, in DAFS we now lock
-individual volumes. Locking an individual volume is done by locking a
-certain byte in the file /vicepX/.volume.lock. To lock volume with ID
-1234, you lock 1 byte at offset 1234 (with VLockFile: fcntl on unix,
-LockFileEx on windows as of the time of this writing). To read-lock the
-volume, acquire a read lock; to write-lock the volume, acquire a write
-lock.
-
-Due to the potentially very large number of volumes attached by the
-fileserver at once, the fileserver does not keep volumes locked the
-entire time they are attached (which would make volume locking
-potentially very slow). Rather, it locks the volume before attaching,
-and unlocks it when the volume has been attached. However, all other
-programs are expected to acquire a volume lock for the entire duration
-they interact with the volume. Whether a read or write lock is obtained
-is determined by the attachment mode, and whether or not the volume in
-question is an RW volume (see VVolLockType()).
-
-These locks are all acquired non-blocking, so we can just fail if we
-fail to acquire a lock. That is, an errant process holding a file-level
-lock cannot cause any process to just hang, waiting for a lock.
-
- -- re-reading volume headers
-
-Since we cannot know whether a volume is writable or not until the
-volume header is read, and we cannot atomically upgrade file-level
-locks, part of attachment can now occur twice (see attach2 and
-attach_volume_header). What occurs is we read the vol header, assuming
-the volume is readonly (acquiring a read or write lock as necessary).
-If, after reading the vol header, we discover that the volume is
-writable and that means we need to acquire a write lock, we read the vol
-header again while acquiring a write lock on the header.
-
- -- verifying checkouts
-
-Since the fileserver does not hold volume locks for the entire time a
-volume is attached, there could have been a potential race between the
-fileserver and other programs. Consider when a non-fileserver program
-checks out a volume from the fileserver via FSSYNC, then locks the
-volume. Before the program locked the volume, the fileserver could have
-restarted and attached the volume. Since the fileserver releases the
-volume lock after attachment, the fileserver and the other program could
-both think they have control over the volume, which is a problem.
-
-To prevent this non-fileserver programs are expected to verify that
-their volume is checked out after locking it (FSYNC_VerifyCheckout).
-What this does is ask the fileserver for the current volume operation on
-the specific volume, and verifies that it matches how the program
-checked out the volume.
-
-For example, programType X checks out volume V from the fileserver, and
-then locks it. We then ask the fileserver for the current volume
-operation on volume V. If the programType on the vol operation does not
-match (or the PID, or the checkout mode, or other things), we know the
-fileserver must have restarted or something similar, and we do not have
-the volume checked out like we thought we did.
-
-If the program determines that the fileserver may have restarted, it
-then must retry checking out and locking the volume (or return an
-error).
git://git.openafs.org / openafs.git / blobdiff