•  


GitHub - tipichris/YAPCache: APC based cache to reduce the database load for the YOURLS URL shortener
Skip to content

APC based cache to reduce the database load for the YOURLS URL shortener

License

Notifications You must be signed in to change notification settings

tipichris/YAPCache

Folders and files

Name Name
Last commit message
Last commit date

Latest commit

 

History

76 Commits
 
 
 
 
 
 
 
 

Repository files navigation

YAPCache

YAPCache is an APC based caching plugin for the YOURLS URL shortener.

YAPCache is designed to remove a lot of the database traffic from YOURLS, primarily the write load from doing the logging and click tracking. We have attempted to strike a balance between keeping most information, but spilling it in some cases in the name of higher performance. YAPCache will write data back to the database based on either the time since the last write or the amount of data currently cached. YAPCache also adds an API call to YOURLS that can be used to force a write out to the database.

YAPCache is a fork of Ian Barber's YOURLS APC Cache , with a few changes listed below . You should not try to install both plugins at the same time!

Installation

  1. If you previously used another APC cache with YOURLS, uninstall it
  2. Download the latest version of YAPCache
  3. Copy the plugins/yapcache folder into your user/plugins folder for YOURLS
  4. Set up the parameters for YAPCache in your YOURLS configuration file, user/config.php ( see below )
  5. Copy the cache.php file into user/
  6. There is no need to activate this plugin (by the same token, deactivating it via the admin panel will not disable it?to do that remove or rename user/cache.php )

A recent version of APC is required.

Operation

Caches

There are four separate caches operated by the plugin:

Keyword Cache : A read cache which caches the keyword -> long URL look up in APC. This is done on request, and by default is cached for about one hour. This period is configurable with YAPC_READ_CACHE_TIMEOUT . This cache will be destroyed if updates are made to the keyword.

Options Cache : A read cache which caches YOURLS options in APC to avoid the need to retrieve these from the database at every request. Cache period is defined by YAPC_READ_CACHE_TIMEOUT and is one hour by default. This cache will be destroyed if any options are updated.

Click caching : A write cache for records of clicks. Rather than writing directly to the database, we write clicks to APC. We keep a record of how long it is since we last wrote clicks to the database and if that period exceeds YAPC_WRITE_CACHE_TIMEOUT seconds (default 120) we write all cached clicks to the database. We also keep an eye on how many URLs we are caching click information for and will write to the database if this figure exceeds YAPC_MAX_UPDATES (default 200). Additionally, if the number of clicks stored for a single URL exceeds YAPC_MAX_CLICKS all clicks are written to the database. Since database writes can involve a lot of updates in quick succession, in either case, if the current server load exceeds YAPC_MAX_LOAD we delay the write. We bundle update queries up in a single transaction, which will reduce the overhead involved considerably as long as your table supports transactions.

Log caching : A write cache similar to the click tracking, but tracking the log entries for each request. Note that each request for the same URL will increase the number of log table records being cached. In contrast, multiple requests for the same URL will not increase the number of records being cached, only the number of clicks recorded in a single record. The consequence of this is that log caching is likely to reach YAPC_MAX_UPDATES faster than click caching.

Flushing the cache with an API call

It is also possible to manually flush the cache out to the database using an API call. If your YOURLS installation is private you will need to authenticate in the usual way. If YAPC_API_USER is defined and not empty only the user defined can flush the cache. To flush you construct an API call in the usual way using the action flushcache . eg

http://your-own-domain-here.com/yourls-api.php?action=flushcache&signature=1002a612b4

The API call is useful if you want to be sure that the cache will be written out at a defined interval even if there are periods when few requests are coming in. You can, for example, call it every five minutes from crontab with something likely

*/5  *  *  *  *  curl -s "https://your-own-domain-here.com/yourls-api.php?action=flushcache&signature=1002a612b4" > /dev/null

You might also consider flushing the cache before restarts of the webserver. Many log rotation scripts, for example, will restart Apache after rotating the log, so it can be useful to use a script to flush the cache immediately before the log is rotated.

It is possible to disable writing to the database as part of a normal request by setting both YAPC_WRITE_CACHE_TIMEOUT and YAPC_MAX_UPDATES to 0. Writes to the database will then only be triggered by the flushcache API call. As database writes can be slow this approach may improve user experience by ensuring that redirects are never delayed by writing data out to the database.

Will you loose clicks?

Almost certainly. Whilst we've taken care to try to minimise this there will be times when clicks and logs cached in APC disappear before they have been written to the database. APC is not ideal for holding volatile data that isn't stored elsewhere yet. If it runs low on memory it will start pruning its cached data and that could mean clicks and logs. Webserver restarts will also clear APC's cache, and on many systems these happen regularly when the logs are rotated. In deciding on suitable values for the various configuration options you will need to balance performance against the risk of loosing data. The longer you cache writes for, the more likely you are to loose data, and the more data you are likely to loose.

If loosing the odd click is unacceptable to you you probably shouldn't use this plugin.

Configuration

The plugin comes with working defaults, but you will probably want to adjust things to your particular needs. The following constants can be defined in user/config.php to alter the plugin's behaviour, eg

define("
YAPC_READ_CACHE_TIMEOUT
", 
1800
);

YAPC_WRITE_CACHE_TIMEOUT

Interger. Default: 120.
Number of seconds to cache data for before writing it out to the database. A value of 0 will disable writing based on time

YAPC_READ_CACHE_TIMEOUT

Interger. Default: 3600.
Number of seconds to cache reads from the database. We do try to delete cached data that has been changed, but not all YOURLS functions that change data have hooks we can use, so it is possible that for a time some stats etc will appear out of date in the admin interface. Redirects should be current, however

YAPC_LONG_TIMEOUT

Interger. Default: 86400.
A timeout used for a few long lasting indexes. You probably don't need to change this

YAPC_LOCK_TIMEOUT

Interger. Default: 30.
Maximum number of seconds to hold a lock on the indexes whilst doing a database write. If things are working this shouldn't take anywhere near 30 seconds (maybe 2 maximum)

YAPC_MAX_LOAD

Interger. Default: 0.7.
If the system load exceeds this value apc-cache will delay writes to the database. A sensible value for this will depend, amongst other things, on how many processors you have. A value of 0 means don't check the load before writing. This setting has no effect on Windows.

When the time cached exceeds YAPC_WRITE_CACHE_HARD_TIMEOUT writes will be done no matter what the load.

YAPC_WRITE_CACHE_HARD_TIMEOUT

Interger. Default: 600.
Number of seconds before a write of cached data to the database is forced, even if the load exceeds YAPC_MAX_LOAD. This setting has no effect if YAPC_WRITE_CACHE_TIMEOUT is set to 0.

YAPC_BACKOFF_TIME

Interger. Default: 30.
Number of seconds to delay the next attempt to write to the database if the load exceeds YAPC_MAX_LOAD

YAPC_MAX_UPDATES

Interger. Default: 200.
The maximum number of updates of each type (clicks and logs) to hold in the cache before writing out to the database. When the number of cached updates exceeds this value they will be written to the database, irrespective of how long they have been cached. However, they won't be written out if the load exceeds YAPC_MAX_LOAD. A value of 0 means never write out on the basis of the number of cached updates.

YAPC_MAX_CLICKS

Interger. Default: 30.
The maximum number of clicks that will be stored for a single URL. If a single URL has had more than this number of clicks since the last time the click cache was written to the database then a write will be performed. However, no write will be done if the load exceeds YAPC_MAX_LOAD. A value of 0 disables this test.

YAPC_API_USER

String. Default: empty string
The name of a user who is allowed to use the flushcache API call to force a write to the database. If set to false, the flushcache API call is disabled. If set to an empty string, any user may force a database write.

YAPC_STATS_SHUNT

String. Default: Undefined.
If set to none this will cause the caching of the clicks and logredirects to be disabled, and the queries logged as normal. This is handy if you want to keep the URL caching, but still have 100% accurate stats (though the benefit of the plugin will be pretty small then).

If set to drop this will cause the clicks and log redirect information to be dropped completely (a more aggressive NOSTATS). There will be no clicks or logredirects logged.

YAPC_SKIP_CLICKTRACK

Boolean. Default: false.
If true this will cause the plugin to take no action on click/redirect logging (if this is being handled by another plugin for example).

YAPC_ID

String. Default: ycache-
A string which is prepended to all APC keys. This is useful if you run two or more instances of YOURLS on the same server and need to avoid their APC cache keys clashing.

YAPC_DEBUG

Boolean. Default: false.
If true additional debug information is sent using PHP's error_log() function. You will find this wherever your server is configured to send PHP errors to.

YAPC_REDIRECT_FIRST

Boolean. Default: false.
Set this to true to send redirects to the client first, and deal with logging and database updates later. This is experimental and highly likely to interact badly with certain other plugins. In particular, it is known to not work with some plugins which change the default HTTP status code to 302. To work around this use the YAPC_REDIRECT_FIRST_CODE setting instead.

If you are potentially caching large numbers of updates, a request that triggers a database write may result in a slow response as normally the database writes are performed first, then a response is sent to the client. This option allows you to respond to the client first and do the slow stuff later. If you are only doing updates based on an API call this setting probably has no benefit and is best avoided.

YAPC_REDIRECT_FIRST_CODE

Interger. Default: 301
The HTTP status code to send with redirects when YAPC_REDIRECT_FIRST is true. Defaults to 301 (moved permanantly). 302 is the most likely alternative (although 303 or 307 are possible). Has no effect if YAPC_REDIRECT_FIRST is false

Difference from Yourls-APC-Cache

The main differences between YAPCache and Ian Barber's original Yourls-APC-Cache are summarised below

  • YAPCache uses a different strategy for caching clicks. Instead of one timer for each URL, YAPCache uses a single timer for all URLs. This is somewhat more aggressive, ie clicks are more likely to be cached. It also means that multiple URLs are updated in the database at the same time. By wrapping these in a transaction the transaction overhead is reduced (ie one transaction for multiple updates, rather than the one transaction per update implied by autocommit)

  • YAPCache uses a different approach to timers. YAPCache writes the time into an APC key and then checks that, rather than relying on the key's TTL. This allows a bit more flexibility in the logic of when to write out to the database, allowing some other changes, including

    • An option to do writes on the basis of the number of records cached as well as / instead of the time since the last write

    • Writes can be delayed if the server load exceeds a threshold

    • YAPCache provides an API call that can be used to trigger a write out to the database.

  • YAPCache includes an experimental option to send the redirect to the client first and delay the slower work of updated the database until afterwards

  • A few minor bugs have been fixed

Which one should you use?

It's your choice. YAPCache has a few extra features?if you need these, use YAPCache. Yourls-APC-Cache is older and has been in use for several years longer. It is more tested (although not without bugs). YAPCache is based on this mature code, but it does include some substantial changes that as yet have had only limited testing in a production environment.

About

APC based cache to reduce the database load for the YOURLS URL shortener

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

- "漢字路" 한글한자자동변환 서비스는 교육부 고전문헌국역지원사업의 지원으로 구축되었습니다.
- "漢字路" 한글한자자동변환 서비스는 전통문화연구회 "울산대학교한국어처리연구실 옥철영(IT융합전공)교수팀"에서 개발한 한글한자자동변환기를 바탕하여 지속적으로 공동 연구 개발하고 있는 서비스입니다.
- 현재 고유명사(인명, 지명등)을 비롯한 여러 변환오류가 있으며 이를 해결하고자 많은 연구 개발을 진행하고자 하고 있습니다. 이를 인지하시고 다른 곳에서 인용시 한자 변환 결과를 한번 더 검토하시고 사용해 주시기 바랍니다.
- 변환오류 및 건의,문의사항은 juntong@juntong.or.kr로 메일로 보내주시면 감사하겠습니다. .
Copyright ⓒ 2020 By '전통문화연구회(傳統文化硏究會)' All Rights reserved.
 한국   대만   중국   일본