Как разбить строку php

mb_str_split

Функция вернёт массив строк, это версия str_split() с поддержкой кодировок переменного размера символов, а также кодировок фиксированного размера из 1, 2 или 4-байтовых символов. Если указан параметр length , строка разбивается на куски указанной длины в символах (не байтах). Может быть указан необязательный параметр encoding , это хорошая практика.

Список параметров

Строка ( string ) для разделения на символы или куски.

Если указано, каждый элемент возвращаемого массива будет состоять из нескольких символов вместо одного символа.

Параметр encoding представляет собой символьную кодировку. Если он опущен или равен null , вместо него будет использовано значение внутренней кодировки.

Строка, указывающая одну из поддерживаемых кодировок.

Возвращаемые значения

mb_str_split() возвращает массив строк.

Список изменений

Версия	Описание
8.0.0	Теперь параметр encoding может принимать значение null .
8.0.0	Функция больше не возвращает false в случае неудачи.

Смотрите также

• str_split() — Преобразует строку в массив

User Contributed Notes 3 notes

1 year ago

if( !function_exists(‘mb_str_split’)) <
function mb_str_split( $string = », $length = 1 , $encoding = null ) <
if(!empty($string)) <
$split = array();
$mb_strlen = mb_strlen($string,$encoding);
for($pi = 0; $pi < $mb_strlen; $pi += $length)<
$substr = mb_substr($string, $pi,$length,$encoding);
if( !empty($substr)) <
$split[] = $substr;
>
>
>
return $split;
>
>

3 years ago

Note: function return NULL if can’t convert argument type.

if (! in_array ( $encoding , mb_list_encodings (), true )) static $aliases ;
if ( $aliases === null ) $aliases = [];
foreach ( mb_list_encodings () as $encoding ) $encoding_aliases = mb_encoding_aliases ( $encoding );
if ( $encoding_aliases ) foreach ( $encoding_aliases as $alias ) $aliases [] = $alias ;
>
>
>
>
if (! in_array ( $encoding , $aliases , true )) trigger_error ( ‘mb_str_split(): Unknown encoding «‘ . $encoding . ‘»‘ , E_USER_WARNING );
return null ;
>
>

$result = [];
$length = mb_strlen ( $string , $encoding );
for ( $i = 0 ; $i < $length ; $i += $split_length ) $result [] = mb_substr ( $string , $i , $split_length , $encoding );
>
return $result ;
>
?>

3 years ago

Lazy polyfill for UTF-8 only:

function utf8_str_split(string $input, int $splitLength = 1)
$re = \sprintf(‘/\\G.+/us’, $splitLength);
\preg_match_all($re, $input, $m);
return $m[0];
>

• Функции для работы с многобайтовыми строками
  • mb_​check_​encoding
  • mb_​chr
  • mb_​convert_​case
  • mb_​convert_​encoding
  • mb_​convert_​kana
  • mb_​convert_​variables
  • mb_​decode_​mimeheader
  • mb_​decode_​numericentity
  • mb_​detect_​encoding
  • mb_​detect_​order
  • mb_​encode_​mimeheader
  • mb_​encode_​numericentity
  • mb_​encoding_​aliases
  • mb_​ereg_​match
  • mb_​ereg_​replace_​callback
  • mb_​ereg_​replace
  • mb_​ereg_​search_​getpos
  • mb_​ereg_​search_​getregs
  • mb_​ereg_​search_​init
  • mb_​ereg_​search_​pos
  • mb_​ereg_​search_​regs
  • mb_​ereg_​search_​setpos
  • mb_​ereg_​search
  • mb_​ereg
  • mb_​eregi_​replace
  • mb_​eregi
  • mb_​get_​info
  • mb_​http_​input
  • mb_​http_​output
  • mb_​internal_​encoding
  • mb_​language
  • mb_​list_​encodings
  • mb_​ord
  • mb_​output_​handler
  • mb_​parse_​str
  • mb_​preferred_​mime_​name
  • mb_​regex_​encoding
  • mb_​regex_​set_​options
  • mb_​scrub
  • mb_​send_​mail
  • mb_​split
  • mb_​str_​pad
  • mb_​str_​split
  • mb_​strcut
  • mb_​strimwidth
  • mb_​stripos
  • mb_​stristr
  • mb_​strlen
  • mb_​strpos
  • mb_​strrchr
  • mb_​strrichr
  • mb_​strripos
  • mb_​strrpos
  • mb_​strstr
  • mb_​strtolower
  • mb_​strtoupper
  • mb_​strwidth
  • mb_​substitute_​character
  • mb_​substr_​count
  • mb_​substr
  • Copyright © 2001-2023 The PHP Group
  • My PHP.net
  • Contact
  • Other PHP.net sites
  • Privacy policy

  explode

  Returns an array of strings, each of which is a substring of string formed by splitting it on boundaries formed by the string separator .

  Parameters

  The boundary string.

  The input string.

  If limit is set and positive, the returned array will contain a maximum of limit elements with the last element containing the rest of string .

  If the limit parameter is negative, all components except the last — limit are returned.

  If the limit parameter is zero, then this is treated as 1.

  Note:

  Prior to PHP 8.0, implode() accepted its parameters in either order. explode() has never supported this: you must ensure that the separator argument comes before the string argument.

  Return Values

  Returns an array of string s created by splitting the string parameter on boundaries formed by the separator .

  If separator is an empty string («»), explode() throws a ValueError . If separator contains a value that is not contained in string and a negative limit is used, then an empty array will be returned, otherwise an array containing string will be returned. If separator values appear at the start or end of string , said values will be added as an empty array value either in the first or last position of the returned array respectively.

  Changelog

  Version	Description
  8.0.0	explode() will now throw ValueError when separator parameter is given an empty string ( «» ). Previously, explode() returned false instead.

  Examples

  Example #1 explode() examples

  // Example 1
  $pizza = «piece1 piece2 piece3 piece4 piece5 piece6» ;
  $pieces = explode ( » » , $pizza );
  echo $pieces [ 0 ]; // piece1
  echo $pieces [ 1 ]; // piece2

  // Example 2
  $data = «foo:*:1023:1000::/home/foo:/bin/sh» ;
  list( $user , $pass , $uid , $gid , $gecos , $home , $shell ) = explode ( «:» , $data );
  echo $user ; // foo
  echo $pass ; // *

  Example #2 explode() return examples

  /*
  A string that doesn’t contain the delimiter will simply
  return a one-length array of the original string.
  */
  $input1 = «hello» ;
  $input2 = «hello,there» ;
  $input3 = ‘,’ ;
  var_dump ( explode ( ‘,’ , $input1 ) );
  var_dump ( explode ( ‘,’ , $input2 ) );
  var_dump ( explode ( ‘,’ , $input3 ) );

  The above example will output:

  array(1) ( [0] => string(5) "hello" ) array(2) ( [0] => string(5) "hello" [1] => string(5) "there" ) array(2) ( [0] => string(0) "" [1] => string(0) "" )

  Example #3 limit parameter examples

  // positive limit
  print_r ( explode ( ‘|’ , $str , 2 ));

  // negative limit
  print_r ( explode ( ‘|’ , $str , — 1 ));
  ?>

  The above example will output:

  Array ( [0] => one [1] => two|three|four ) Array ( [0] => one [1] => two [2] => three )

  Notes

  Note: This function is binary-safe.

  See Also

  • preg_split() — Split string by a regular expression
  • str_split() — Convert a string to an array
  • mb_split() — Split multibyte string using regular expression
  • str_word_count() — Return information about words used in a string
  • strtok() — Tokenize string
  • implode() — Join array elements with a string

  User Contributed Notes 4 notes

  1 year ago

  Note that an empty input string will still result in one element in the output array. This is something to remember when you are processing unknown input.

  For example, maybe you are splitting part of a URI by forward slashes (like «articles/42/show» => [«articles», «42», «show»]). And maybe you expect that an empty URI will result in an empty array («» => []). Instead, it will contain one element, with an empty string:

  $uri = » ;
  $parts = explode ( ‘/’ , $uri );
  var_dump ( $parts );

  2 years ago

  Be careful, while most non-alphanumeric data types as input strings return an array with an empty string when used with a valid separator, true returns an array with the string «1»!

  var_dump(explode(‘,’, null)); //array(1) < [0]=>string(0) «» >
  var_dump(explode(‘,’, false)); //array(1) < [0]=>string(0) «» >

  var_dump(explode(‘,’, true)); //array(1) < [0]=>string(1) «1» >

  10 days ago

  If your data is smaller than the expected count with the list expansion:

  $data = «foo:*:1023:1000::/home/foo:/bin/sh» ;
  list( $user , $pass , $uid , $gid , $gecos , $home , $shell , $nu ) = explode ( «:» , $data );
  ?>

  The result is a warning not an error:

  PHP Warning: Undefined array key 7 in .

  The solution is to pad the array to the expected length:

  $data = «foo:*:1023:1000::/home/foo:/bin/sh» ;
  list( $user , $pass , $uid , $gid , $gecos , $home , $shell , $nu ) = array_pad ( explode ( «:» , $data ), 8 , «» );
  // where 8 is the count of the list arguments
  ?>

  1 year ago

  If you want to directly take a specific value without having to store it in another variable, you can implement the following:

  echo $status_only = explode(‘-‘, $status)[0];

  • Copyright © 2001-2023 The PHP Group
  • My PHP.net
  • Contact
  • Other PHP.net sites
  • Privacy policy

  preg_split

  Если указан, функция возвращает не более, чем limit подстрок. Оставшаяся часть строки будет возвращена в последней подстроке. Специальное значение limit , равное -1 или 0, подразумевает отсутствие ограничения.

  flags может быть любой комбинацией следующих флагов (объединённых с помощью побитового оператора | ): PREG_SPLIT_NO_EMPTY Если указан этот флаг, функция preg_split() вернёт только непустые подстроки. PREG_SPLIT_DELIM_CAPTURE Если указан этот флаг, выражение, заключённое в круглые скобки в разделяющем шаблоне, также извлекается из заданной строки и возвращается функцией. PREG_SPLIT_OFFSET_CAPTURE

  Если указан этот флаг, для каждой найденной подстроки будет указана её позиция в исходной строке. Необходимо помнить, что этот флаг меняет формат возвращаемого массива: каждый элемент будет содержать массив, содержащий в индексе с номером 0 найденную подстроку, а смещение этой подстроки в параметре subject — в индексе 1 .

  Возвращаемые значения

  Возвращает массив, состоящий из подстрок заданной строки subject , которая разбита по границам, соответствующим шаблону pattern или false в случае возникновения ошибки.

  Ошибки

  Если переданный шаблон регулярного выражения не компилируется в допустимое регулярное выражение, выдаётся ошибка уровня E_WARNING .

  Примеры

  Пример #1 preg_split() пример: Получение подстрок из заданного текста

  // разбиваем строку по произвольному числу запятых и пробельных символов,
  // которые включают в себя » «, \r, \t, \n и \f
  $keywords = preg_split ( «/[\s,]+/» , «hypertext language, programming» );
  print_r ( $keywords );
  ?>

  Результат выполнения данного примера:

  Array ( [0] => hypertext [1] => language [2] => programming )

  Пример #2 Разбиваем строку на составляющие символы

  $str = ‘string’ ;
  $chars = preg_split ( ‘//’ , $str , — 1 , PREG_SPLIT_NO_EMPTY );
  print_r ( $chars );
  ?>

  Результат выполнения данного примера:

  Array ( [0] => s [1] => t [2] => r [3] => i [4] => n [5] => g )

  Пример #3 Разбиваем строку с указанием смещения для каждой из найденных подстрок

  $str = ‘hypertext language programming’ ;
  $chars = preg_split ( ‘/ /’ , $str , — 1 , PREG_SPLIT_OFFSET_CAPTURE );
  print_r ( $chars );
  ?>

  Результат выполнения данного примера:

  Array ( [0] => Array ( [0] => hypertext [1] => 0 ) [1] => Array ( [0] => language [1] => 10 ) [2] => Array ( [0] => programming [1] => 19 ) )

  Примечания

  Подсказка

  Если вам не нужна мощь регулярных выражений, вы можете выбрать более быстрые (хоть и простые) альтернативы наподобие explode() или str_split() .

  Подсказка

  Если соответствий не нашлось, то возвращается массив с единственным элементом равным всей строке.

  Смотрите также

  • «Регулярные выражения PCRE»
  • preg_quote() — Экранирует символы в регулярных выражениях
  • implode() — Объединяет элементы массива в строку
  • preg_match() — Выполняет проверку на соответствие регулярному выражению
  • preg_match_all() — Выполняет глобальный поиск шаблона в строке
  • preg_replace() — Выполняет поиск и замену по регулярному выражению
  • preg_last_error() — Возвращает код ошибки выполнения последнего регулярного выражения PCRE

  User Contributed Notes 18 notes

  14 years ago

  Sometimes PREG_SPLIT_DELIM_CAPTURE does strange results.

  $content = ‘Lorem ipsum dolor sit amet consectetuer.’ ;
  $chars = preg_split ( ‘/<[^>]*[^\/]>/i’ , $content , — 1 , PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE );
  print_r ( $chars );
  ?>
  Produces:
  Array
  (
  [0] => Lorem ipsum dolor
  [1] => sit amet
  [2] => consec
  [3] => tet
  [4] => uer
  )

  So that the delimiter patterns are missing. If you wanna get these patters remember to use parentheses.

  

  $chars = preg_split ( ‘/(<[^>]*[^\/]>)/i’ , $content , — 1 , PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE );
  print_r ( $chars ); //parentheses added
  ?>
  Produces:
  Array
  (
  [0] =>
  [1] => Lorem ipsum dolor
  [2] =>
  [3] => sit amet
  [4] =>
  [5] => consec
  [6] =>
  [7] => tet
  [8] =>
  [9] => uer
  [10] =>
  [11] => .
  )

  14 years ago

  Extending m.timmermans’s solution, you can use the following code as a search expression parser:

  $search_expression = «apple bear \»Tom Cruise\» or ‘Mickey Mouse’ another word» ;
  $words = preg_split ( «/[\s,]*\\\»([^\\\»]+)\\\»[\s,]*|» . «[\s,]*'([^’]+)'[\s,]*|» . «[\s,]+/» , $search_expression , 0 , PREG_SPLIT_NO_EMPTY | PREG_SPLIT_DELIM_CAPTURE );
  print_r ( $words );
  ?>

  The result will be:
  Array
  (
  [0] => apple
  [1] => bear
  [2] => Tom Cruise
  [3] => or
  [4] => Mickey Mouse
  [5] => another
  [6] => word
  )

  1. Accepted delimiters: white spaces (space, tab, new line etc.) and commas.

  2. You can use either simple (‘) or double («) quotes for expressions which contains more than one word.

  8 years ago

  This regular expression will split a long string of words into an array of sub-strings, of some maximum length, but only on word-boundries.

  I use the reg-ex with preg_match_all(); but, I’m posting this example here (on the page for preg_split()) because that’s where I looked when I wanted to find a way to do this.

  Hope it saves someone some time.

  // example of a long string of words
  $long_string = ‘Your IP Address will be logged with the submitted note and made public on the PHP manual user notes mailing list. The IP address is logged as part of the notes moderation process, and won\’t be shown within the PHP manual itself.’ ;

  // «word-wrap» at, for example, 60 characters or less
  $max_len = 60 ;

  // this regular expression will split $long_string on any sub-string of
  // 1-or-more non-word characters (spaces or punctuation)
  if( preg_match_all ( «/. >(?=\W+)/» , $long_string , $lines ) !== False )

  // $lines now contains an array of sub-strings, each will be approx.
  // $max_len characters — depending on where the last word ended and
  // the number of ‘non-word’ characters found after the last word
  for ( $i = 0 ; $i < count ( $lines [ 0 ]); $i ++) echo "[ $i ] < $lines [ 0 ][ $i ]>\n» ;
  >
  >
  ?>

  4 years ago

  Assuming you’re using UTF-8, this function can be used to separate Unicode text into individual codepoints without the need for the multibyte extension.

  preg_split ( ‘//u’ , $text , — 1 , PREG_SPLIT_NO_EMPTY );

  ?>

  The words «English», «Español», and «Русский» are all seven letters long. But strlen would report string lengths 7, 8 and 14, respectively. The preg_split above would return a seven-element array in all three cases.

  It splits ‘한국어’ into the array [‘한’, ‘국’, ‘어’] instead of the 9-character array that str_split($text) would produce.

  12 years ago

  Here is another way to split a CamelCase string, which is a simpler expression than the one using lookaheads and lookbehinds:

  preg_split(‘/([[:upper:]][[:lower:]]+)/’, $last, null, PREG_SPLIT_DELIM_CAPTURE|PREG_SPLIT_NO_EMPTY)

  It makes the entire CamelCased word the delimiter, then returns the delimiters (PREG_SPLIT_DELIM_CAPTURE) and omits the empty values between the delimiters (PREG_SPLIT_NO_EMPTY)

  13 years ago

  If you want to split by a char, but want to ignore that char in case it is escaped, use a lookbehind assertion.

  In this example a string will be split by «:» but «\:» will be ignored:

  19 years ago

  To clarify the «limit» parameter and the PREG_SPLIT_DELIM_CAPTURE option,

  $preg_split ( ‘(/ /)’ , ‘1 2 3 4 5 6 7 8’ , 4 , PREG_SPLIT_DELIM_CAPTURE );
  ?>

  returns:

  (‘1’, ‘ ‘, ‘2’, ‘ ‘ , ‘3’, ‘ ‘, ‘4 5 6 7 8’)

  So you actually get 7 array items not 4

  18 years ago

  preg_split() behaves differently from perl’s split() if the string ends with a delimiter. This perl snippet will print 5:

  my @a = split(/ /, «a b c d e «);
  print scalar @a;

  The corresponding php code prints 6:

  This is not necessarily a bug (nowhere does the documentation say that preg_split() behaves the same as perl’s split()) but it might surprise perl programmers.

  12 years ago

  You must be caution when using lookbehind to a variable match.
  For example:
  ‘/(? to match a new line when not \ is before it don’t go as spected as it match \r as the lookbehind (becouse isn’t a \) and is optional before \n.

  You must use this for example:
  ‘/((?That match a alone \n (not preceded by \r or \) or a \r\n not preceded by a \.

  2 years ago

  Beware that it is not safe to assume there are no empty values returned by PREG_SPLIT_NO_EMPTY, nor that you will see no delimiters if you use PREG_SPLIT_DELIM_CAPTURE, as there are some edge cases where these are not true.

  # As expected, splitting a string by itself returns two empty strings:
  var_export ( preg_split ( «/x/» , «x» ));

  # But if we add PREG_SPLIT_NO_EMPTY, then instead of an empty array, we get the delimiter.
  var_export ( preg_split ( «/x/» , «x» , PREG_SPLIT_NO_EMPTY ));

  And if we try to split an empty string , then instead of an empty array, we get an empty string even with PREG_SPLIT_NO_EMPTY .
  var_export ( preg_split ( «/x/» , «» , PREG_SPLIT_NO_EMPTY ));

  9 years ago

  This is a function to truncate a string of text while preserving the whitespace (for instance, getting an excerpt from an article while maintaining newlines). It will not jive well with HTML, of course.

  /**
  * Truncates a string of text by word count
  * @param string $text The text to truncate
  * @param int $max_words The maximum number of words
  * @return string The truncated text
  */
  function limit_words ( $text , $max_words ) $split = preg_split ( ‘/(\s+)/’ , $text , — 1 , PREG_SPLIT_DELIM_CAPTURE );
  $truncated = » ;
  for ( $i = 0 ; $i < min ( count ( $split ), $max_words * 2 ); $i += 2 ) $truncated .= $split [ $i ]. $split [ $i + 1 ];
  >
  return trim ( $truncated );
  >
  ?>

  14 years ago

  To split a camel-cased string using preg_split() with lookaheads and lookbehinds:

  14 years ago

  If the task is too complicated for preg_split, preg_match_all might come in handy, since preg_split is essentially a special case.

  I wanted to split a string on a certain character (asterisk), but only if it wasn’t escaped (by a preceding backslash). Thus, I should ensure an even number of backslashes before any asterisk meant as a splitter. Look-behind in a regular expression wouldn’t work since the length of the preceding backslash sequence can’t be fixed. So I turned to preg_match_all:

  // split a string at unescaped asterisks
  // where backslash is the escape character
  $splitter = «/\\*((?:[^\\\\*]|\\\\.)*)/» ;
  preg_match_all ( $splitter , «* $string » , $aPieces , PREG_PATTERN_ORDER );
  $aPieces = $aPieces [ 1 ];

  // $aPieces now contains the exploded string
  // and unescaping can be safely done on each piece
  foreach ( $aPieces as $idx => $piece )
  $aPieces [ $idx ] = preg_replace ( «/\\\\(.)/s» , «$1» , $piece );
  ?>

  12 years ago

  Limit = 1 may be confusing. The important thing is that in case of limit equals to 1 will produce only ONE substring. Ergo the only one substring will be the first one as well as the last one. Tnat the rest of the string (after the first delimiter) will be placed to the last substring. But last is the first and only one.

  $output = $preg_split ( ‘(/ /)’ , ‘1 2 3 4 5 6 7 8’ , 1 );

  echo $output [ 0 ] //will return whole string!;

  $output = $preg_split ( ‘(/ /)’ , ‘1 2 3 4 5 6 7 8’ , 2 );

  echo $output [ 0 ] //will return 1;
  echo $output [ 1 ] //will return ‘2 3 4 5 6 7 8’;

  strtok

  Альтернативная сигнатура (не поддерживается с именованными аргументами):

  strtok ( string $token ): string | false

  strtok() разбивает строку string на подстроки (токены), используя в качестве разделителей символы из token . Например, строку «This is an example string» можно разбить на отдельные слова, используя пробел в качестве разделителя ( token ).

  Заметьте, что исходная строка ( string ) передаётся только при первом вызове этой функции. Последующим вызовам передаются только разделители ( token ), так как эта функция сохраняет исходную строку и запоминает позицию в этой строке между вызовами. Для работы с новой строкой ( string ) нужно снова вызвать функцию с двумя аргументами. Обратите внимание, что в параметре token можно использовать несколько разделителей. Строка будет делиться по любому найденному символу, присутствующему в параметре ( token ).

  Замечание:

  Функция ведёт себя немного иначе, чем можно было бы ожидать, знакомясь с explode() . Во-первых, последовательность из двух или более смежных символов token в анализируемой строке считается одним разделителем. Также игнорируется token , расположенный в начале или конце строки. Например, если используется строка «;aaa;;bbb;» , последовательные вызовы strtok() с «;» в качестве token вернут строки «aaa» и «bbb», а затем false . В результате строка будет разделена только на два элемента, а explode(«;», $string) вернёт массив из 5 элементов.

  Список параметров

  Строка ( string ), разбиваемая на подстроки (токены).

  Разделитель строки string .

  Возвращаемые значения

  Токен в виде строки ( string ) или false , если токенов больше нет.

  Примеры

  Пример #1 Пример использования strtok()

  $string = «This is\tan example\nstring» ;
  /* В качестве разделителей используем пробел, табуляцию и перевод строки */
  $tok = strtok ( $string , » \n\t» );

  while ( $tok !== false ) echo «Word= $tok
  » ;
  $tok = strtok ( » \n\t» );
  >
  ?>

  Пример #2 Способ обработки пустых подстрок функцией strtok()

  $first_token = strtok ( ‘/something’ , ‘/’ );
  $second_token = strtok ( ‘/’ );
  var_dump ( $first_token , $second_token );
  ?>

  Результат выполнения данного примера:

  string(9) "something" bool(false)

  Пример #3 Различие между strtok() и explode()

  $parts = [];
  $tok = strtok ( $string , «;» );
  while ( $tok !== false ) $parts [] = $tok ;
  $tok = strtok ( «;» );
  >
  echo json_encode ( $parts ), «\n» ;

  $parts = explode ( «;» , $string );
  echo json_encode ( $parts ), «\n» ;

  Результат выполнения данного примера:

  ["aaa","bbb"] ["","aaa","","bbb",""]

  Примечания

  Внимание

  Эта функция может возвращать как логическое значение false , так и значение не типа boolean, которое приводится к false . За более подробной информацией обратитесь к разделу Булев тип. Используйте оператор === для проверки значения, возвращаемого этой функцией.

  Смотрите также

  • explode() — Разбивает строку с помощью разделителя

  User Contributed Notes 20 notes

  10 years ago

  // strtok example
  $str = ‘Hello to all of Ukraine’ ;
  echo strtok ( $str , ‘ ‘ ). ‘ ‘ . strtok ( ‘ ‘ ). ‘ ‘ . strtok ( ‘ ‘ );
  ?>
  Result:
  Hello to all

  19 years ago

   /** get leading, trailing, and embedded separator tokens that were 'skipped'
  if for some ungodly reason you are using php to implement a simple parser that 
  needs to detect nested clauses as it builds a parse tree */
  

  $seps = ‘()’ ;
  $tok = strtok ( $str , $seps ); // return false on empty string or null
  $cur = 0 ;
  $dumbDone = FALSE ;
  $done = ( FALSE === $tok );
  while (! $done ) // process skipped tokens (if any at first iteration) (special for last)
  $posTok = $dumbDone ? strlen ( $str ) : strpos ( $str , $tok , $cur );
  $skippedMany = substr ( $str , $cur , $posTok — $cur ); // false when 0 width
  $lenSkipped = strlen ( $skippedMany ); // 0 when false
  if ( 0 !== $lenSkipped ) $last = strlen ( $skippedMany ) — 1 ;
  for( $i = 0 ; $i <= $last ; $i ++)$skipped = $skippedMany [ $i ];
  $cur += strlen ( $skipped );
  echo «skipped: $skipped \n» ;
  >
  >
  if ( $dumbDone ) break; // this is the only place the loop is terminated

  // process current tok
  echo «curr tok: » . $tok . «\n» ;

  // update cursor
  $cur += strlen ( $tok );

  // get any next tok
  if (! $dumbDone ) $tok = strtok ( $seps );
  $dumbDone = ( FALSE === $tok );
  // you’re not really done till you check for trailing skipped
  >
  >;
  ?>

  12 years ago

  If you have memory-usage critical solution, you should keep in mind, that strtok function holds input string parameter (or reference to it?) in memory after usage.

  function tokenize ( $str , $token_symbols ) $word = strtok ( $str , $token_symbols );
  while ( false !== $word ) // do something here.
  $word = strtok ( $token_symbols );
  >
  >
  ?>
  Test-cases with handling ~10MB plain-text file:
  Case #1 — unset $str variable
  $token_symbols = » \t\n» ;
  $str = file_get_contents ( ’10MB.txt’ ); // mem usage 9.75383758545 MB (memory_get_usage() / 1024 / 1024));
  tokenize ( $str , $token_symbols ); // mem usage 9.75400161743 MB
  unset( $str ); // 9.75395584106 MB
  ?>
  Case #1 result: memory is still used

  Case #2 — call strtok again
  $token_symbols = » \t\n» ;
  $str = file_get_contents ( ’10MB.txt’ ); // 9.75401306152 MB
  tokenize ( $str , $token_symbols ); // 9.75417709351
  strtok ( » , » ); // 9.75421524048
  ?>
  Case #2 result: memory is still used

  Case #3 — call strtok again AND unset $str variable
  $token_symbols = » \t\n» ;
  $str = file_get_contents ( ’10MB.txt’ ); // 9.75410079956 MB
  tokenize ( $str , $token_symbols ); // 9.75426483154 MB
  unset( $str );
  strtok ( » , » ); // 0.0543975830078 MB
  ?>
  Case #3 result: memory is free

  So, better solution for tokenize function:
  function tokenize ( $str , $token_symbols , $token_reset = true ) $word = strtok ( $str , $token_symbols );
  while ( false !== $word ) // do something here.
  $word = strtok ( $token_symbols );
  >

  8 years ago
  14 years ago

  Simple way to tokenize search parameters, including double or single quoted keys. If only one quote is found, the rest of the string is assumed to be part of that token.

  $token = strtok ( $keywords , ‘ ‘ );
  while ( $token ) <
  // find double quoted tokens
  if ( $token < 0 >== ‘»‘ ) < $token .= ' ' . strtok ( '"' ). '"' ; >
  // find single quoted tokens
  if ( $token < 0 >== «‘» )

  $tokens [] = $token ;
  $token = strtok ( ‘ ‘ );
  >
  ?>

  Use substr(1,strlen($token)) and remove the part that adds the trailing quotes if you want your output without quotes.

  14 years ago

  This looks very simple, but it took me a long time to figure out so I thought I’d share it incase someone else was wanting the same thing:

  this should work similar to substr() but with tokens instead!

  /* subtok(string,chr,pos,len)
  *
  * chr = chr used to seperate tokens
  * pos = starting postion
  * len = length, if negative count back from right
  *
  * subtok(‘a.b.c.d.e’,’.’,0) = ‘a.b.c.d.e’
  * subtok(‘a.b.c.d.e’,’.’,0,2) = ‘a.b’
  * subtok(‘a.b.c.d.e’,’.’,2,1) = ‘c’
  * subtok(‘a.b.c.d.e’,’.’,2,-1) = ‘c.d’
  * subtok(‘a.b.c.d.e’,’.’,-4) = ‘b.c.d.e’
  * subtok(‘a.b.c.d.e’,’.’,-4,2) = ‘b.c’
  * subtok(‘a.b.c.d.e’,’.’,-4,-1) = ‘b.c.d’
  */
  function subtok ( $string , $chr , $pos , $len = NULL ) return implode ( $chr , array_slice ( explode ( $chr , $string ), $pos , $len ));
  >
  ?>

  explode breaks the tokens up into an array, array slice alows you to pick then tokens you want, and then implode converts it back to a string

  although its far from a clone, this was inspired by mIRC’s gettok() function

  9 years ago

  Might be pointing out the obvious but if you’d rather use a for loop rather than a while (to keep the token strings on the same line for readability for example), it can be done. Added bonus, it doesn’t put a $tok variable outside the loop itself either.
  Downside however is that you’re not able to manually free up the memory used using the technique mentioned by elarlang.

  for( $tok = strtok ( $str , ‘ _-.’ ); $tok !== false ; $tok = strtok ( ‘ _-.’ ))
  echo » $tok
  » ;
  >
  ?>

  9 years ago

  If you want to tokenize by only one letter, explode() is much faster compared to strtok().

  $str = str_repeat ( ‘foo ‘ , 10000 );

  //explode()
  $time = microtime ( TRUE );
  $arr = explode ( $str , ‘ ‘ );
  $time = microtime ( TRUE )- $time ;
  echo «explode(): $time sec.» . PHP_EOL ;

  //strtok()
  $time = microtime ( TRUE );
  $ret = strtok ( ‘ ‘ , $str );
  while( $ret !== FALSE ) $ret = strtok ( ‘ ‘ );
  >
  $time = microtime ( TRUE )- $time ;
  echo «strtok(): $time sec.» . PHP_EOL ;

  ?>

  The result is : (PHP 5.3.3 on CentOS)

  explode():0.001317024230957 sec.
  strtok():0.0058917999267578 sec.

  explode() is about five times fast in short strings, too.

  11 years ago

  Note that strtok may receive different tokens each time. Therefore, if, for example, you wish to extract several words and then the rest of the sentence:

  $text = «13 202 5 This is a long message explaining the error codes.» ;
  $error1 = strtok ( $text , » » ); //13
  $error2 = strtok ( » » ); //202
  $error3 = strtok ( » » ); //5
  $error_message = strtok ( «» ); //Notice the different token parameter
  echo $error_message ; //This is a long message explaining the error codes.
  ?>

  14 years ago

  As of the change in strtok()’s handling of empty strings, it is now useless for scripts that rely on empty data to function.

  Take for instance, a standard header. (with UNIX newlines)

  http/1.0 200 OK\n
  Content-Type: text/html\n
  \n
  —HTML BODY HERE—

  When parsing this with strtok, one would wait until it found an empty string to signal the end of the header. However, because strtok now skips empty segments, it is impossible to know when the header has ended.
  This should not be called `correct’ behavior, it certainly is not. It has rendered strtok incapable of (properly) processing a very simple standard.

  This new functionality, however, does not affect Windows style headers. You would search for a line that only contains «\r»
  This, however, is not a justification for the change.

  14 years ago

  Here is a java like StringTokenizer class using strtok function:

  /**
  * The string tokenizer class allows an application to break a string into tokens.
  *
  * @example The following is one example of the use of the tokenizer. The code:
  *
* * $str = 'this is:@\t\n a test!';
* $delim = ' !@:'\t\n; // remove these chars
* $st = new StringTokenizer($str, $delim);
* while ($st->hasMoreTokens()) * echo $st->nextToken() . "\n";
* >
* prints the following output:
* this
* is
* a
* test
* ?>
*
  */
  class StringTokenizer

  /**
  * @var string
  */
  private $delim ;
  /**
  * Constructs a string tokenizer for the specified string
  * @param string $str String to tokenize
  * @param string $delim The set of delimiters (the characters that separate tokens)
  * specified at creation time, default to ‘ ‘
  */
  public function __construct ( /*string*/ $str , /*string*/ $delim = ‘ ‘ ) $this -> token = strtok ( $str , $delim );
  $this -> delim = $delim ;
  >

  public function __destruct () unset( $this );
  >

  /**
  * Tests if there are more tokens available from this tokenizer’s string. It
  * does not move the internal pointer in any way. To move the internal pointer
  * to the next element call nextToken()
  * @return boolean — true if has more tokens, false otherwise
  */
  public function hasMoreTokens () return ( $this -> token !== false );
  >

  /**
  * Returns the next token from this string tokenizer and advances the internal
  * pointer by one.
  * @return string — next element in the tokenized string
  */
  public function nextToken () $current = $this -> token ;
  $this -> token = strtok ( $this -> delim );
  return $current ;
  >
  >
  ?>

  7 months ago

  Hello, portuguese documentation of strtok is wrong, at this part which the example(2) is wrong.

  Exemplo #2 Comportamento antigo da strtok()
  $first_token = strtok ( ‘/something’ , ‘/’ );
  $second_token = strtok ( ‘/’ );
  var_dump ( $first_token , $second_token );
  ?>

  O exemplo acima produzirá:

  string(0) «»
  string(9) «something»

  (this example above, should be inverted as this:)

  Correct:
  string(9) «something»
  string(0) «»

  (exemple 3 is correct)
  Exemplo #3 Novo comportamento da strtok()
  $first_token = strtok ( ‘/something’ , ‘/’ );
  $second_token = strtok ( ‘/’ );
  var_dump ( $first_token , $second_token );
  ?>

  O exemplo acima produzirá:

  string(9) «something»
  bool(false)

  4 years ago

  I found this useful for parsing user entered links in text fields.

  2 years ago

  After obtaining zero or more tokens with calls to strtok, you can obtain the remainder of the input string by calling strtok with an empty string as the delimiter.

  7 years ago

  @maisuma you invert paramaters of explode() and strtok() functions, your code does not do what you expect.
  You expect to read the input string token after token so equivalent code for strtok() is arra_filter(explode()) because explode() return lines of empty string when several delimiters are contiguous in the read string, for example 2 whitespaces between.

  In fact strtok() is much faster (x2 at least) than arra_filter(explode()) if the read string contains several contiguous delimiters ,
  it is slower if the read string contains one and only one delimiter between tokens.

  $repeat = 10 ;
  $delimiter = ‘:’ ;
  $str = str_repeat ( ‘foo:’ , $repeat );

  $timeStrtok = microtime ( TRUE );
  $token = strtok ( $str , $delimiter );
  while( $token !== FALSE ) //echo $token . ‘,’;
  $token = strtok ( $delimiter );
  >
  $timeStrtok -= microtime ( TRUE );

  $timeExplo = microtime ( TRUE );
  $arr = explode ( $delimiter , $str );
  //$arr = array_filter($arr);
  $timeExplo -= microtime ( TRUE );

  $X = 1000000 ; $unit = ‘microsec’ ;

  echo PHP_EOL . ‘ explode() : ‘ . — $timeExplo . ‘ ‘ . $unit . PHP_EOL . ‘ strtok() : ‘ . — $timeStrtok . ‘ ‘ . $unit . PHP_EOL ;

  $timeExplo = round (- $timeExplo * $X );
  $timeStrtok = round (- $timeStrtok * $X );

  echo PHP_EOL . ‘ explode() : ‘ . $timeExplo . ‘ ‘ . $unit . PHP_EOL . ‘ strtok() : ‘ . $timeStrtok . ‘ ‘ . $unit . PHP_EOL ;
  echo ‘ ratio explode / strtok : ‘ . round ( $timeExplo / $timeStrtok , 1 ) . PHP_EOL ;

  12 years ago

  Here’s a simple class that allows you to iterate through string tokens using a foreach loop.

  /**
  * The TokenIterator class allows you to iterate through string tokens using
  * the familiar foreach control structure.
  *
  * Example:
  *
* * $string = 'This is a test.';
* $delimiters = ' ';
* $ti = new TokenIterator($string, $delimiters);
*
* foreach ($ti as $count => $token) <
* echo sprintf("%d, %s\n", $count, $token);
* >
*
* // Prints the following output:
* // 0. This
* // 1. is
* // 2. a
* // 3. test.
*
  */
  class TokenIterator implements Iterator
  <
  /**
  * The string to tokenize.
  * @var string
  */
  protected $_string ;

  /**
  * The token delimiters.
  * @var string
  */
  protected $_delims ;

  /**
  * Stores the current token.
  * @var mixed
  */
  protected $_token ;

  /**
  * Internal token counter.
  * @var int
  */
  protected $_counter = 0 ;

  /**
  * Constructor.
  *
  * @param string $string The string to tokenize.
  * @param string $delims The token delimiters.
  */
  public function __construct ( $string , $delims )
  <
  $this -> _string = $string ;
  $this -> _delims = $delims ;
  $this -> _token = strtok ( $string , $delims );
  >

  /**
  * @see Iterator::current()
  */
  public function current ()
  <
  return $this -> _token ;
  >

  /**
  * @see Iterator::key()
  */
  public function key ()
  <
  return $this -> _counter ;
  >

  /**
  * @see Iterator::next()
  */
  public function next ()
  <
  $this -> _token = strtok ( $this -> _delims );

  if ( $this -> valid ()) <
  ++ $this -> _counter ;
  >
  >

  /**
  * @see Iterator::rewind()
  */
  public function rewind ()
  <
  $this -> _counter = 0 ;
  $this -> _token = strtok ( $this -> _string , $this -> _delims );
  >

  /**
  * @see Iterator::valid()
  */
  public function valid ()
  <
  return $this -> _token !== FALSE ;
  >
  >
  ?>

  9 years ago
  17 years ago

  This function takes a string and returns an array with words (delimited by spaces), also taking into account quotes, doublequotes, backticks and backslashes (for escaping stuff).
  So

  $string = «cp ‘my file’ to `Judy’s file`»;
  var_dump(parse_cli($string));

  Way it works, runs through the string character by character, for each character looking up the action to take, based on that character and its current $state.
  Actions can be (one or more of) adding the character/string to the current word, adding the word to the output array, and changing or (re)storing the state.
  For example a space will become part of the current ‘word’ (or ‘token’) if $state is ‘doublequoted’, but it will start a new token if $state was ‘unquoted’.
  I was later told it’s a «tokeniser using a finite state automaton». Who knew 🙂

  #_____________________
  # parse_cli($string) /
  function parse_cli ( $string ) $state = ‘space’ ;
  $previous = » ; // stores current state when encountering a backslash (which changes $state to ‘escaped’, but has to fall back into the previous $state afterwards)
  $out = array(); // the return value
  $word = » ;
  $type = » ; // type of character
  // array[states][chartypes] => actions
  $chart = array(
  ‘space’ => array( ‘space’ => » , ‘quote’ => ‘q’ , ‘doublequote’ => ‘d’ , ‘backtick’ => ‘b’ , ‘backslash’ => ‘ue’ , ‘other’ => ‘ua’ ),
  ‘unquoted’ => array( ‘space’ => ‘w ‘ , ‘quote’ => ‘a’ , ‘doublequote’ => ‘a’ , ‘backtick’ => ‘a’ , ‘backslash’ => ‘e’ , ‘other’ => ‘a’ ),
  ‘quoted’ => array( ‘space’ => ‘a’ , ‘quote’ => ‘w ‘ , ‘doublequote’ => ‘a’ , ‘backtick’ => ‘a’ , ‘backslash’ => ‘e’ , ‘other’ => ‘a’ ),
  ‘doublequoted’ => array( ‘space’ => ‘a’ , ‘quote’ => ‘a’ , ‘doublequote’ => ‘w ‘ , ‘backtick’ => ‘a’ , ‘backslash’ => ‘e’ , ‘other’ => ‘a’ ),
  ‘backticked’ => array( ‘space’ => ‘a’ , ‘quote’ => ‘a’ , ‘doublequote’ => ‘a’ , ‘backtick’ => ‘w ‘ , ‘backslash’ => ‘e’ , ‘other’ => ‘a’ ),
  ‘escaped’ => array( ‘space’ => ‘ap’ , ‘quote’ => ‘ap’ , ‘doublequote’ => ‘ap’ , ‘backtick’ => ‘ap’ , ‘backslash’ => ‘ap’ , ‘other’ => ‘ap’ ));
  for ( $i = 0 ; $i <= strlen ( $string ); $i ++) $char = substr ( $string , $i , 1 );
  $type = array_search ( $char , array( ‘space’ => ‘ ‘ , ‘quote’ => ‘\» , ‘doublequote’ => ‘»‘ , ‘backtick’ => ‘`’ , ‘backslash’ => ‘\\’ ));
  if (! $type ) $type = ‘other’ ;
  if ( $type == ‘other’ ) // grabs all characters that are also ‘other’ following the current one in one go
  preg_match ( «/[ \’\»\`\\\]/» , $string , $matches , PREG_OFFSET_CAPTURE , $i );
  if ( $matches ) $matches = $matches [ 0 ];
  $char = substr ( $string , $i , $matches [ 1 ]- $i ); // yep, $char length can be > 1
  $i = $matches [ 1 ] — 1 ;
  >else // no more match on special characters, that must mean this is the last word!
  // the .= hereunder is because we *might* be in the middle of a word that just contained special chars
  $word .= substr ( $string , $i );
  break; // jumps out of the for() loop
  >
  >
  $actions = $chart [ $state ][ $type ];
  for( $j = 0 ; $j < strlen ( $actions ); $j ++) $act = substr ( $actions , $j , 1 );
  if ( $act == ‘ ‘ ) $state = ‘space’ ;
  if ( $act == ‘u’ ) $state = ‘unquoted’ ;
  if ( $act == ‘q’ ) $state = ‘quoted’ ;
  if ( $act == ‘d’ ) $state = ‘doublequoted’ ;
  if ( $act == ‘b’ ) $state = ‘backticked’ ;
  if ( $act == ‘e’ ) < $previous = $state ; $state = 'escaped' ; >
  if ( $act == ‘a’ ) $word .= $char ;
  if ( $act == ‘w’ ) < $out [] = $word ; $word = '' ; >
  if ( $act == ‘p’ ) $state = $previous ;
  >
  >
  if ( strlen ( $word )) $out [] = $word ;
  return $out ;
  >

  Похожие публикации:

  1. Как изменить ttl на mac os
  2. Как понять в какой строке ошибка java
  3. Как посимвольно считать строку в java
  4. Как узнать тип строки python